{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_Linear_Regression",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Linear Regression"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\" width=150>\n",
        "\n",
        "In this lesson we will learn about linear regression. We will first understand the basic math behind it and then implement it in Python. We will also look at ways of interpreting the linear model.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Overview"
      ]
    },
    {
      "metadata": {
        "id": "-qHciBsX93ej",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/linear.png\" width=250>\n",
        "\n",
        "$\\hat{y} = XW$\n",
        "\n",
        "*where*:\n",
        "* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n",
        "* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n",
        "* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ "
      ]
    },
    {
      "metadata": {
        "id": "QAgr7Grv9pb6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* **Objective:**  Use inputs $X$ to predict the output $\\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted and target outcomes. Training data $(X, y)$ is used to train the model and learn the weights $W$ using stochastic gradient descent (SGD).\n",
        "* **Advantages:**\n",
        "  * Computationally simple.\n",
        "  * Highly interpretable.\n",
        "  * Can account for continuous and categorical features.\n",
        "* **Disadvantages:**\n",
        "  * The model will perform well only when the data is linearly separable (for classification).\n",
        "  * Usually not used for classification and only for regression.\n",
        "* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuos regression tasks only.\n"
      ]
    },
    {
      "metadata": {
        "id": "xP7XD24-09Io",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Training"
      ]
    },
    {
      "metadata": {
        "id": "476yPgTM1BKJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*Steps*: \n",
        "1. Randomly initialize the model's weights $W$.\n",
        "2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n",
        "3. Compare the predictions $\\hat{y}$ with the actual target values $y$ with the objective (cost) function to determine loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it. (the $\\frac{1}{2}$ is just for convenicing the derivative operation).\n",
        "  * $MSE = J(\\theta) = \\frac{1}{2}\\sum_{i}(\\hat{y}_i - y_i)^2$\n",
        "4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights.\n",
        "  * $J(\\theta) = \\frac{1}{2}\\sum_{i}(\\hat{y}_i - y_i)^2 = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 $\n",
        "  * $\\frac{\\partial{J}}{\\partial{W}} = X(\\hat{y} - y)$\n",
        "4. Apply backpropagation to update the weights $W$ using a learning rate $\\alpha$ and an optimization technique (ie. stochastic gradient descent). The simplified intuition is that the gradient tells you the direction for how to increase something so subtracting it will help you go the other way since we want to decrease loss $J(\\theta)$.\n",
        "  * $W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "5. Repeat steps 2 - 4 until model performs well."
      ]
    },
    {
      "metadata": {
        "id": "jvJKjkMeJP4Q",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Data"
      ]
    },
    {
      "metadata": {
        "id": "RuPl9qlSJTIY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We're going to create some simple dummy data to apply linear regression on."
      ]
    },
    {
      "metadata": {
        "id": "HRXD7LqVJZ43",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from argparse import Namespace\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NFsKg-Z6IWqG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Arguments\n",
        "args = Namespace(\n",
        "    seed=1234,\n",
        "    data_file=\"sample_data.csv\",\n",
        "    num_samples=100,\n",
        "    train_size=0.75,\n",
        "    test_size=0.25,\n",
        "    num_epochs=100,\n",
        ")\n",
        "\n",
        "# Set seed for reproducability\n",
        "np.random.seed(args.seed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NWux2lcoIWss",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Generate synthetic data\n",
        "def generate_data(num_samples):\n",
        "    X = np.array(range(num_samples))\n",
        "    y = 3.65*X + 10\n",
        "    return X, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2mb2SjSQIWvF",
        "colab_type": "code",
        "outputId": "3aa66ef6-3c88-40fd-f53a-a77f93c8e052",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# Generate random (linear) data\n",
        "X, y = generate_data(args.num_samples)\n",
        "data = np.vstack([X, y]).T\n",
        "df = pd.DataFrame(data, columns=['X', 'y'])\n",
        "df.head()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>10.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>13.65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>17.30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "      <td>20.95</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.0</td>\n",
              "      <td>24.60</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     X      y\n",
              "0  0.0  10.00\n",
              "1  1.0  13.65\n",
              "2  2.0  17.30\n",
              "3  3.0  20.95\n",
              "4  4.0  24.60"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "6LwVVOkiLfBN",
        "colab_type": "code",
        "outputId": "ab2ecddb-bbeb-4117-d1cf-29312846da16",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        }
      },
      "cell_type": "code",
      "source": [
        "# Scatter plot\n",
        "plt.title(\"Generated data\")\n",
        "plt.scatter(x=df[\"X\"], y=df[\"y\"])\n",
        "plt.show()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAFZCAYAAABJ+lxSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X10lPWd///XJJNhGDIhmWFiQaV1\nrShKIKHRnyioCQkBrRUUKM2iUtHiQltZqIiIFGtbBCy2FPagIEhlu6RGbekpGg4FXPZ7IIumJxvY\n7kE9xxYRSCaEJORmIMP1+8MyBszNzOSaZG6ej3N6DplcueaTN4OvXu/5XO+xGIZhCAAA9Kmkvl4A\nAAAgkAEAiAoEMgAAUYBABgAgChDIAABEAQIZAIAoQCADJjMMQ7/5zW/0rW99S5MmTVJhYaEee+wx\nHT58uK+XJknav3+/Pvvss5B/7sYbb9Snn35qynFer1d//vOfQ14DEM8IZMBkL730kv70pz9p06ZN\neuedd/Tuu+9q/Pjx+u53v6vTp0/39fL02muvhRXIZiovL9eePXv6dA1AtCGQAROdOXNGW7du1cqV\nK5WZmSlJSk5O1owZM7R37165XC5J0smTJ/X444+rqKhIRUVFeu+99yRJn376qcaOHavf/OY3uvfe\nezVu3Djt3LlT0udX3uvWrVNRUZHy8vL005/+VH6/X5L04IMP6qWXXtKkSZNUUVEhr9er2bNna+LE\nicrPz9eWLVskSb/85S918OBBPfnkk9q5c6fOnTunn/70pyoqKlJ+fr42bNgQ+F3ee+89FRYWatKk\nSdq0aVOnv3NXx61fv15FRUUqKCjQnDlz1NDQoCNHjugnP/mJysrK9K//+q+SpDfeeEOTJk3ShAkT\n9M///M86fvy4GX8dQGwxAJhm3759RlFRUbfHPfTQQ8ZLL71kGIZhfPLJJ8Ytt9xinD592jh27Jhx\n4403Gq+//rphGIaxc+dOo7Cw0DAMw3j77beNe+65x2hoaDDOnz9vfO973wscN3PmTOORRx4x/H6/\nYRiG8ZOf/MRYtmyZYRiG8fe//9246aabjM8++8wwDMPIy8szDh06ZBiGYaxbt854+OGHDZ/PZzQ1\nNRmTJ0829uzZY7S1tRm33367sX//fsMwDOPVV181hg0bZhw7duyS36Or46qqqowxY8YYjY2Nht/v\nN2bNmmWsX7/eMAzDWLt2rbFkyRLDMAzD6/UaI0aMME6cOGEYhmEsXrw48D0gkXCFDJiovr4+cBUs\nSQ0NDZo4caImTpyoO+64Qxs3blRzc7PKy8s1a9YsSdJXv/pVfeMb3whcJbe1ten++++XJN10002B\n9vLevXv1wAMPyOl0ymq1atq0adq1a1fgue68804lJX3+T3rp0qV69tlnJUlXX321PB5Ph+/r7t27\nV8XFxbLZbHI4HLrvvvu0a9cuffLJJzp37pzGjh0rSZoyZUqHv29Xx40YMUL79u1TamqqkpKSlJOT\no2PHjn3pHG63Wx988IG+8pWvSJJyc3M7PA6Id9a+XgAQT1wul6qrqwNfp6Wl6d1335UkPfPMM2pt\nbVVjY6MMw9CMGTMCxzU3N+vWW2+V9HmL2+FwSJKSkpJ04cIFSVJjY6NeffVVlZSUSJL8fv8l4T9w\n4MDAn6uqqvSLX/xCJ06cUFJSkmpqagLnaa+xsVErVqzQmjVrJEnnzp3TyJEjVV9fr9TU1A7P3V5X\nx7W0tGjFihUqLy8PHHvXXXd96Rx+v19r167Vnj175Pf71dTUpGuuuabD5wPiGYEMmCg7O1u1tbX6\n3//9X914440dHuN2u5WcnKw333xTAwYMuOR7Xe1OzszMVH5+vmbOnNntOp588kk9/PDD+s53viOL\nxaJx48Z1es5HHnlEeXl5lzz+8ccf6+zZs4GvO9uMNnDgwE6P27p1qz755BO99dZbGjBggF566SWd\nOnXqS+fYuXOn9uzZo23btsnlcul3v/ud/vjHP3b7OwLxhpY1YKLU1FTNnTtXixYt0t/+9jdJ0oUL\nF/SnP/1J77zzjoYOHSqr1ao777xT27dvl/T5leTTTz+tEydOdHnu8ePH6w9/+INaWlokSdu3b9fb\nb7/d4bG1tbUaMWKELBaL3n77bbW0tKi5uVmSZLVa1djYGDjnG2+8Ib/fL8Mw9G//9m/6z//8Tw0d\nOlTJycmBq9u33npLFovlS8/T1XG1tbX6p3/6Jw0YMEDHjx/Xe++91+EaamtrdeWVV8rlcqmurk7v\nvPOOmpqagqw4ED8IZMBkjz32mB5++GH98Ic/1MSJE1VQUKC33npLa9eu1X333SdJWr58uQ4dOqSJ\nEydqypQpuvrqqzV48OAuz1tQUKC8vDxNmTJFEydO1J49ewLv3V7uiSee0Lx583TvvfequblZ3/72\nt/Xss8/q73//u4qKirRgwQJt2bJFxcXFGjJkiO655x5NnDhRH3/8sb7xjW8oJSVFzz//vJYsWaJJ\nkybJYrEE2ujtdXXcjBkzdOjQIRUVFWnlypVavHixDhw4oNdee0233367Dh48qAceeEDf/OY3debM\nGRUWFmrhwoWaP3++Tp48qRdeeKGHfxNAbLEYBp+HDABAX+MKGQCAKEAgAwAQBQhkAACiAIEMAEAU\nIJABAIgCfToYpKam0dTzZWQ4VFfXbOo5ExF1NAd1NAd1NAd1NEdP6+jxODv9XlxdIVutyX29hLhA\nHc1BHc1BHc1BHc0RyTrGVSADABCrCGQAAKIAgQwAQBQgkAEAiAIEMgAAUYBABgAgChDIAABEAQIZ\nAIAO+M77VV3XLN95f688X59O6gIAINr4L1xQyZ6P9JejNTrd4JMrrZ9yhnn07fyvR/R5uUIGAKCd\nkj0faff7n6q2wSdDUm2DT7vf/1Qlez6K6PNyhQwAgD5vUdfUNesvR2s6/P5fjnrVeq4tYs9PIAMA\nElr7FnVtg6/T4+oaW1XX4ItYcBLIAICEdrFF3Z0Mp10Zaf3UWN8SkXUQyACAhNRdi/pyOcMGyW6z\nytwPDv4CgQwASCjBtqglyWKRXE67coYNivguawIZAJBQgm1Ru5z9NH/6KHnS+6tfSuQ/T5pABgAk\nhFBb1KOv9+gqT2qEV/UFAhkAENeitUV9OQIZABCXfOf9qj/rU9mhY9pbcbzb43u7RX05AhkAEFcu\nvyJOsgT3c73dor4cgQwAiCuXb9q6YHR+bF+2qC9HIAMA4kKom7b6ukV9OQIZABDTQtm01V5ft6gv\nRyADAGJasPcVJ1kkQ9HTor5ct4Hc0tKixYsXq7a2Vj6fT3PnzlVZWZmOHDmi9PR0SdLs2bN11113\naceOHdq6dauSkpI0ffp0TZs2LeK/AAAgMYXaor4ze4iKbhmqgan9oqJFfbluA3nv3r0aMWKEHnvs\nMR0/flyPPPKIcnJytGDBAuXl5QWOa25u1vr161VaWqqUlBRNnTpVhYWFgdAGAMAMPbmvODkpqZdW\nGbpuA/nuu+8O/PnEiRO64oorOjyusrJSWVlZcjqdkqTRo0eroqJC+fn5Ji0VAIDoHX3ZU0G/hzxj\nxgydPHlSGzZs0GuvvaZt27Zpy5YtcrvdevbZZ+X1euVyuQLHu1wu1dR03UbIyHDIajW3SB6P09Tz\nJSrqaA7qaA7qaI5YrmPruTbVNfjksFv1Px/XBvUzY7OvVM6Ng01fS6TqGHQgb9++XX/961/15JNP\nasmSJUpPT9fw4cP1yiuvaN26dcrJybnkeMPo4savf6iraw59xV3weJyqqYnUB2MlDupoDupoDupo\njlitY/v29OkGnwam2nTm7LlOj2/for53zFDTf+ee1rGrMO82kA8fPiy3263Bgwdr+PDh8vv9GjZs\nmNxutyQpPz9fy5cvV1FRkbxeb+DnqqurlZ2dHfaiAQCJq7Oxl12Fcay1qC/X7bvb77//vjZv3ixJ\n8nq9am5u1rJly3Ts2DFJUnl5ua677jqNGjVKVVVVamhoUFNTkyoqKpSbmxvZ1QMA4or/wgX9dvdR\nLd14UItfPqj3/tL9DOqLLt5XHIthLAVxhTxjxgw988wzKi4uVmtrq5YtWyaHw6H58+erf//+cjgc\nWrFihex2uxYuXKjZs2fLYrFo3rx5gQ1eAAAEI5Sxlxmp/VTf5FNGlN5XHCqLEcybvRESbb19fI46\nmoM6moM6miPa63jxnuJflf5PUNO23Gl2LZuVqxZfW6/eV9yn7yEDABAp4Y69zBk2SE6HTU6HLYKr\n610EMgCgz8TL2EszEMgAgF4Xb2MvzUAgAwB6TbyOvTQDgQwA6DXxOvbSDAQyACDiQm1RR9tnFfcG\nAhkAEDE9aVEnGgIZAGC6zkZfdiYRW9SXI5ABAKa5/Io4yRLczyVii/pyBDIAwDShjL5M9Bb15Qhk\nAECPhbppixb1lxHIAICwhTv6khb1lxHIAICwMfrSPAQyACBkjL40H4EMAAgaoy8jh0AGAASN0ZeR\nQyADALrF6MvII5ABAJ1i9GXvIZABAF/C6MveRyADAAIYfdl3CGQAQACjL/sOgQwACe5ie7p/Pyuj\nL/sQgQwACap9e/p0g08DU206c/ZcUD9Li9p8BDIAJKjL29NdhTGjLyOPQAaABBPqPcUSoy97A4EM\nAAkilHuKM1L7qb7JpwxGX/YaAhkAEkSwYy/daXYtm5WrFl8bV8S9iEAGgDjnO+/XJycagm5R5wwb\nJKfDJqfDFuGVob1uA7mlpUWLFy9WbW2tfD6f5s6dqxtuuEGLFi2S3++Xx+PR6tWrZbPZtGPHDm3d\nulVJSUmaPn26pk2b1hu/AwCgA4y9jC3dBvLevXs1YsQIPfbYYzp+/LgeeeQRjR49WsXFxZo0aZLW\nrFmj0tJSTZ48WevXr1dpaalSUlI0depUFRYWKj09vTd+DwDAPzD2MjZ1G8h333134M8nTpzQFVdc\nofLycj333HOSpLy8PG3evFnXXHONsrKy5HQ6JUmjR49WRUWF8vPzI7R0AEB7jL2MbUG/hzxjxgyd\nPHlSGzZs0He/+13ZbJ+/t+B2u1VTUyOv1yuXyxU43uVyqaYm+C31AICeYexlbAs6kLdv366//vWv\nevLJJ2UYX/wtt/9ze5093l5GhkNWq7ntEY/Haer5EhV1NAd1NAd17FrruTadrG1W5ce1QR3vSbdr\n2aO36ivuAbLb2Nsbqki9Hrv9mzh8+LDcbrcGDx6s4cOHy+/3a8CAAWptbZXdbtepU6eUmZmpzMxM\neb3ewM9VV1crOzu7y3PX1TX3/Ddox+Nxqqam0dRzJiLqaA7qaA7q2LlQNm21N+rrgzTAmqTG+hZR\n2dD09PXYVZh3e5f3+++/r82bN0uSvF6vmpubddttt6msrEyStGvXLo0bN06jRo1SVVWVGhoa1NTU\npIqKCuXm5oa9aABA1y62qLsL4yTL5/9zp9lVkHsVLeoo1e0V8owZM/TMM8+ouLhYra2tWrZsmUaM\nGKGnnnpKJSUlGjJkiCZPnqyUlBQtXLhQs2fPlsVi0bx58wIbvAAA5gl19OWd2UP0nYk3yn/uPLuo\no5jFCObN3ggxuw1Fa8sc1NEc1NEc1PELPbmv+CtXDKSOJohky5p38wEgRgQ7+pL7imMTgQwAUS7U\nFjX3FccmAhkAohSjLxMLgQwAUYbRl4mJQAaAKMHoy8RGIANAlGD0ZWIjkAGgj4W6aYsWdXwikAGg\nj4Q7+pIWdXwikAGgjwR7X3GSRTJEizreEcgA0MvCGX1ZdMtQDUztR4s6jhHIANBLenJfcXJSt58F\nhBhHIANAL2H0JbpCIANABF0c8tG/n5XRl+gSgQwAEdC+PX26waeBqTadOXuu0+O5rxgEMgCYqLOx\nl12FMS1qSAQyAJgi3LGXEi1qfI5ABgAThDL2MiO1n+qbfMqgRY12CGQA6IFQ7yl2p9m1bFauWnxt\n3FeMSxDIABCGcMde5gwbJKfDJqfDFsHVIRYRyAAQBsZewmwEMgCEgLGXiBQCGQCCwNhLRBqBDABB\nYOwlIo1ABoAuhNqi5p5ihItABoAO9KRFDYSDQAaAdjobfdkZWtQwC4EMAAp/9CUtapiFQAYAhTb6\nkhY1IoFABpDQQt20RYsakRJUIK9atUoffPCB2traNGfOHO3Zs0dHjhxRenq6JGn27Nm66667tGPH\nDm3dulVJSUmaPn26pk2bFtHFA0C4wh19SYsakdJtIB88eFAffvihSkpKVFdXpylTpujWW2/VggUL\nlJeXFziuublZ69evV2lpqVJSUjR16lQVFhYGQhsAogmjLxFtug3km2++WSNHjpQkpaWlqaWlRX6/\n/0vHVVZWKisrS06nU5I0evRoVVRUKD8/3+QlA0D4GH2JaNVtICcnJ8vhcEiSSktLdccddyg5OVnb\ntm3Tli1b5Ha79eyzz8rr9crlcgV+zuVyqaYmuBc8AEQaoy8R7YLe1LV7926VlpZq8+bNOnz4sNLT\n0zV8+HC98sorWrdunXJyci453jC62KL4DxkZDlmt5v4/To/Haer5EhV1NAd1NIcZddz4+6qgWtSe\ndLuWPXqrvuIeILstvva98no0R6TqGNSrbf/+/dqwYYM2bdokp9OpMWPGBL6Xn5+v5cuXq6ioSF6v\nN/B4dXW1srOzuzxvXV1zmMvumMfjVE1No6nnTETU0RzU0Rw9rePFFvX/q+x+yIckjfr6IA2wJqmx\nvkXx9LfH69EcPa1jV2HebR+msbFRq1at0ssvvxzYoPWDH/xAx44dkySVl5fruuuu06hRo1RVVaWG\nhgY1NTWpoqJCubm5YS8aAHrCf+GCfrv7qJZuPKhlmw912aa2WCR3ml0FuVexaQt9ptsr5J07d6qu\nrk7z588PPHb//fdr/vz56t+/vxwOh1asWCG73a6FCxdq9uzZslgsmjdvXmCDFwD0FkZfIlZZjGDe\n7I0Qs9sntGTMQR3NQR3NEWwdOxp92dW0rYsKcq9SccEwE1Ya3Xg9miOSLev42rEAIGEx+hKxjkAG\nENMYfYl4QSADiEmMvkS8IZABxCRGXyLeEMgAYsbFHdT9+1kZfYm4QyADiHrt29OnG3wamGrTmbPn\nOj2e0ZeIRQQygKjlO+/XCW+T/mP3h5fcU9xVGLNpC7GKQAYQdTq6pzhYbNpCrCKQAUSdUO4pzkjt\np/omnzLYtIUYRyADiBqh3lPsTrNr2axctfja2LSFmEcgA+hz4d5TnDNskJwOm5wOWwRXB/QOAhlA\nn+OeYoBABtCHQm1Rc08x4hmBDKDXhdKi5p5iJAoCGUCvC7ZF7XL203NzxshqGFwRI+7xfzUB9Brf\neb8+rW4MukU9+nqPvjZ4IGGMhMAVMoCI60mLGkgUBDKAiLn4YRBlh45dMvqyM4y9RCIjkAGYLtzR\nl4y9RCIjkAGYLpTRl7Sogc8RyABME+p9xbSogS8QyAB6LNzRl7SogS8QyAB6jNGXQM8RyADCxuhL\nwDwEMoCQMfoSMB+BDCBkoYy+ZNMWEBwCGUDQQm1Rs2kLCB6BDKBbjL4EIo9ABtApRl8CvSeoQF61\napU++OADtbW1ac6cOcrKytKiRYvk9/vl8Xi0evVq2Ww27dixQ1u3blVSUpKmT5+uadOmRXr9ACKA\n0ZdA7+s2kA8ePKgPP/xQJSUlqqur05QpUzRmzBgVFxdr0qRJWrNmjUpLSzV58mStX79epaWlSklJ\n0dSpU1VYWKj09PTe+D0AmIjRl0Dv6zaQb775Zo0cOVKSlJaWppaWFpWXl+u5556TJOXl5Wnz5s26\n5pprlJWVJafTKUkaPXq0KioqlJ+fH8HlAzDLxfZ0/35WRl8CfaDbQE5OTpbD4ZAklZaW6o477tB/\n/dd/yWazSZLcbrdqamrk9XrlcrkCP+dyuVRT0/U/6owMh6xWc/8RezxOU8+XqKijOWKhjn7/BW3+\n4xEdPHxCNWdalOHsp9NBjr8cm32lcm4cHOEVxkYdYwF1NEek6hj0pq7du3ertLRUmzdv1oQJEwKP\nG0bHvazOHm+vrq452KcPisfjVE1No6nnTETU0RyxUsff7j56SXu6qzC+fPTlvWOGRvx3jJU6Rjvq\naI6e1rGrMA8qkPfv368NGzZo06ZNcjqdcjgcam1tld1u16lTp5SZmanMzEx5vd7Az1RXVys7Ozvs\nRQOIrFDvKZYYfQlEUrcz7BobG7Vq1Sq9/PLLgQ1at912m8rKyiRJu3bt0rhx4zRq1ChVVVWpoaFB\nTU1NqqioUG5ubmRXDyBk/gsX9NvdR7V040Et23yoy/uKM1L7KckiudPsKsi9SsWFw5SZ4SCMgQjo\n9gp5586dqqur0/z58wOPvfDCC1q6dKlKSko0ZMgQTZ48WSkpKVq4cKFmz54ti8WiefPmBTZ4AYge\nwY69dKfZtWxWrlp8bVwRA73AYgTzZm+EmP1+Bu+RmIM6miPa6nixRf2r0v8J6jOLC3KvUnHBsF5Y\nWdeirY6xijqao8/fQwYQuxh7CcQGAhmIU4y9BGILgQzEGcZeArGJQAbiDGMvgdhEIANxItT7imlR\nA9GFQAZiXCibttqjRQ1EFwIZiHHB3ld8+dhLWtRAdCGQgRgVaouasZdAdCOQgRjTk/uKk5O6nZYL\noI8QyECMCbZFzaYtILYQyECMCLVFzaYtILYQyECUY/QlkBgIZCBKMfoSSCwEMhBlGH0JJCYCGYgy\njL4EEhOBDEQJRl8CiY1ABvoYoy8BSAQy0OcYfQlAIpCBPsPoSwDtEchAL2P0JYCOEMhAL2P0JYCO\nEMhAL7g45KN/PyujLwF0iEAGIqh9e/p0g08DU206c/Zcp8dzXzGQuAhkIAJ85/064W3Sf+z+8JKx\nl12FMS1qILERyICJwh17KdGiBhIdgQyYKJSxlxmp/VTf5FMGLWoAIpABU4R6T7E7za5ls3LV4mvj\nvmIAkghkoEfCHXuZM2yQnA6bnA5bBFcHIJYENWXg6NGjKigo0LZt2yRJixcv1r333qsHH3xQDz74\noPbt2ydJ2rFjhx544AFNmzZNb7zxRsQWDUSLiy3q7sI4yfL5Dmp3ml0FuVfRngbwJd1eITc3N+v5\n55/XmDFjLnl8wYIFysvLu+S49evXq7S0VCkpKZo6daoKCwuVnp5u/qqBPsbYSwBm6zaQbTabNm7c\nqI0bN3Z5XGVlpbKysuR0OiVJo0ePVkVFhfLz881ZKRAFGHsJIFK6DWSr1Sqr9cuHbdu2TVu2bJHb\n7dazzz4rr9crl8sV+L7L5VJNTXBXD0CsCGXs5XNzxshqGFwRAwhKWJu67rvvPqWnp2v48OF65ZVX\ntG7dOuXk5FxyjGF0cb/HP2RkOGS1mvsfK4/Haer5EhV1vFTruTadrG1W5ce1QR0/NvtKfW3wwAiv\nKnHwejQHdTRHpOoYViC3fz85Pz9fy5cvV1FRkbxeb+Dx6upqZWdnd3meurrmcJ6+Ux6PUzU1jaae\nMxFRxy+E26K+d8xQSaKOJuD1aA7qaI6e1rGrMA/rTa0f/OAHOnbsmCSpvLxc1113nUaNGqWqqio1\nNDSoqalJFRUVys3NDW/FQB/znferuq5Zv939YVC7qF3OfnrukVv008f+PxUXDOP9YgAh6/YK+fDh\nw1q5cqWOHz8uq9WqsrIyzZw5U/Pnz1f//v3lcDi0YsUK2e12LVy4ULNnz5bFYtG8efMCG7yAWBHu\n6EvGXgLoKYsRzJu9EWJ2+4SWjDkSuY6/3X00qE1bUve7qBO5jmaijuagjuaIZMuaSV2AQr+vmE9m\nAmA2AhkJLdzRl7SoAZiNQEZCC/a+4iSLZOjSFjUAmIlARkJi9CWAaEMgI6Ew+hJAtCKQkVBCGX3J\npi0AvYlARkIItUXNpi0AvY1ARlzrSYsaAHoTgYy45DvvV/1Zn8oOHdPeiuPdHk+LGkBfI5ARVxh9\nCSBWEciIK5dv2rrQxWBYWtQAogmBjJh3sT3dv5+V0ZcAYhaBjJjVvj19usGngak2nTl7LqifpUUN\nINoQyIhZl7enuwpjRl8CiHYEMmJOqPcUS4y+BBD9CGTEjFDuKc5I7af6Jp8yGH0JIEYQyIgZwY69\ndKfZtWxWrlp8bVwRA4gZBDKiXqgt6pxhg+R02OR02CK8MgAwD4GMqMXYSwCJhEBG1GHsJYBERCAj\najD2EkAiI5ARNRh7CSCREcjoc6Fu2qJFDSAeEcjoM6Fs2mqPFjWAeEQgo88Ee18xYy8BJAICGb0u\n1BY1Yy8BJAICGb2mJ/cVM/YSQLwjkNFrgm1Rs2kLQCIikBFxobao2bQFIBEFFchHjx7V3LlzNWvW\nLM2cOVMnTpzQokWL5Pf75fF4tHr1atlsNu3YsUNbt25VUlKSpk+frmnTpkV6/YhijL4EgOB1G8jN\nzc16/vnnNWbMmMBja9euVXFxsSZNmqQ1a9aotLRUkydP1vr161VaWqqUlBRNnTpVhYWFSk9Pj+gv\ngOjD6EsACF23O2VsNps2btyozMzMwGPl5eUaP368JCkvL08HDhxQZWWlsrKy5HQ6ZbfbNXr0aFVU\nVERu5Yg6/gsX9NvdR7V040Etfvmg3vtL92EsfdGiJowBJLJur5CtVqus1ksPa2lpkc32+Ufbud1u\n1dTUyOv1yuVyBY5xuVyqqQnuPUPEB0ZfAkD4erypyzA6/q9uZ4+3l5HhkNVq7lWRx+M09XyJKpQ6\ntp5r08naZlV+XBvcudPtWvborfqKe4DstvjeV8jr0RzU0RzU0RyRqmNY/zV0OBxqbW2V3W7XqVOn\nlJmZqczMTHm93sAx1dXVys7O7vI8dXXN4Tx9pzwep2pqGk09ZyIKto7hjr4c9fVBGmBNUmN9i+L5\nb4vXozmoozmoozl6WseuwjysaQu33XabysrKJEm7du3SuHHjNGrUKFVVVamhoUFNTU2qqKhQbm5u\neCtGTLjYou4ujJMsn7eo3Wl2FeReRYsaADrQ7RXy4cOHtXLlSh0/flxWq1VlZWV68cUXtXjxYpWU\nlGjIkCGaPHmyUlJStHDhQs2ePVsWi0Xz5s2T00l7JB4x+hIAzGcxgnmzN0LMbp/QkjFHZ3Vk9GVo\neD2agzqagzqaI5It6/jeUQNTMfoSACKHQEaXLg756N/PyuhLAIggAhkdat+ePt3g08BUm86cPdfp\n8dxXDAA9QyDjEr7zfp3wNumRz5c7AAALt0lEQVQ/dn94ydjLrsKYFjUA9ByBDElf3rCVZAn+Z2lR\nA0DPEciQFNrYy4zUfqpv8imDFjUAmIZATnCh3lPsTrNr2axctfjauK8YAExEICeocMde5gwbJKfD\nJqfDFsHVAUDiIZATVLD3FCdZJEPsoAaASCOQEwxjLwEgOhHICYKxlwAQ3QjkBBHK2Mvn5oyR1TC4\nIgaAXsSlT5zznffr0+rGkMZefm3wQMIYAHoZV8hxqictagBA7yOQ48zFD4MoO3TsktGXnWHsJQBE\nBwI5ToQ7+pKxlwAQHQjkOBHK6Eta1AAQfQjkGBfqfcW0qAEgOhHIMSrc0Ze0qAEgOhHIMYrRlwAQ\nXwjkGMPoSwCITwRyjGD0JQDENwI5RoQy+pJNWwAQewjkKBdqi5pNWwAQmwjkKMXoSwBILARylGH0\nJQAkJgI5SjD6EgASG4EcJRh9CQCJLaxALi8v1xNPPKHrrrtOkjRs2DA9+uijWrRokfx+vzwej1av\nXi2bzWbqYuPNxfZ0/35WRl8CQIIL+wr5lltu0dq1awNfP/300youLtakSZO0Zs0alZaWqri42JRF\nxpv27enTDT4NTLXpzNlzQf0sLWoAiE+mTYwoLy/X+PHjJUl5eXk6cOCAWaeOOxfb07UNPhlSl2Gc\nZPm8Re1Os6sg9ypa1AAQp8K+Qv7oo4/0+OOPq76+Xt///vfV0tISaFG73W7V1ATXgk0kod5TLDH6\nEgASRViB/LWvfU3f//73NWnSJB07dkwPPfSQ/H5/4PuG0cWOpHYyMhyyWs0NGY/Haer5zOD3X9Dm\nPx7RwcMnVF3X0uWx7oF21TW0alB6f906YrAeufcmJSf3/ujLaKxjLKKO5qCO5qCO5ohUHcMK5Cuu\nuEJ33323JGno0KEaNGiQqqqq1NraKrvdrlOnTikzM7Pb89TVNYfz9J3yeJyqqWk09Zw9Eeo9xe40\nu5Y9nKsWX1vgivj06aZeWOmloq2OsYo6moM6moM6mqOndewqzMMK5B07dqimpkazZ89WTU2Namtr\ndf/996usrEz33Xefdu3apXHjxoW94FgX7j3FOcMGyemwyelgdzoAJJqwAjk/P18/+tGP9Oc//1nn\nz5/X8uXLNXz4cD311FMqKSnRkCFDNHnyZLPXGjO4pxgAEKqwAjk1NVUbNmz40uNbtmzp8YJiWaib\ntrinGABwEZO6TBDKB0G0xz3FAICLCGQTBPtZxUkWyRAtagDAlxHIPRBqi5p7igEAnSGQw9CTzypO\nTur9e4oBANGPQA5DsC1qNm0BAIJFIIcg1BY1m7YAAMEikIPQkxY1AADBIJC7EOroS1rUAIBwEcgd\nCHf0JS1qAEC4COQOMPoSANDbCOR2GH0JAOgrBLIYfQkA6HsEshh9CQDoewkdyIy+BABEi4QMZEZf\nAgCiTUIGMqMvAQDRJqECmdGXAIBolRCBzOhLAEC0i9tAvjj2cmBqP7353se0qAEAUS3uArn91fDp\nBp8ynDY1+/xB/SwtagBAX4m7QL58w9bpxnNdHk+LGgAQDeImkH3n/frkREPQG7YkWtQAgOgR84Ec\n7thLiRY1ACB6xHwgB3tPsd2WrAF2q+oafcqgRQ0AiDIxHci+8/6gW9RjRw7WA3deG9h5TYsaABBN\nYjqQ68/6dLqLNnVHYy8zMxy9uEIAAIIT04E8MLWfXGn9OnzvmA1bAIBYEtOflNAvJVk5wzwdfu/i\nhi3CGAAQC2L6CllSYGPWX456VdfYyoYtAEBMMj2Qf/7zn6uyslIWi0VLlizRyJEjzX6KSyQnJam4\nYJgeuPNaJdtS5D93nqtiAEDMMbVl/d///d/629/+ppKSEv3sZz/Tz372MzNP36V+KckaPGgAYQwA\niEmmBvKBAwdUUFAgSbr22mtVX1+vs2fPmvkUAADEJVMD2ev1KiMjI/C1y+VSTU3woywBAEhUEd3U\nZRhGl9/PyHDIajW3xezxOE09X6KijuagjuagjuagjuaIVB1NDeTMzEx5vd7A19XV1fJ4Or4tSZLq\n6prNfHp5PE7V1DSaes5ERB3NQR3NQR3NQR3N0dM6dhXmprasb7/9dpWVlUmSjhw5oszMTKWm8uEN\nAAB0x9Qr5NGjR+umm27SjBkzZLFY9OMf/9jM0wMAELdMfw/5Rz/6kdmnBAAg7sX06EwAAOKFxehu\nKzQAAIg4rpABAIgCBDIAAFGAQAYAIAoQyAAARAECGQCAKEAgAwAQBSL64RK96ec//7kqKytlsVi0\nZMkSjRw5sq+XFDNWrVqlDz74QG1tbZozZ46ysrK0aNEi+f1+eTwerV69Wjabra+XGRNaW1v1zW9+\nU3PnztWYMWOoYxh27NihTZs2yWq16oc//KGuv/566hiipqYmPfXUU6qvr9f58+c1b948eTweLV++\nXJJ0/fXX67nnnuvbRUa5o0ePau7cuZo1a5ZmzpypEydOdPg63LFjh7Zu3aqkpCRNnz5d06ZNC/9J\njThQXl5ufO973zMMwzA++ugjY/r06X28othx4MAB49FHHzUMwzBOnz5t3HnnncbixYuNnTt3GoZh\nGL/4xS+Mf//3f+/LJcaUNWvWGPfff7/x5ptvUscwnD592pgwYYLR2NhonDp1yli6dCl1DMPrr79u\nvPjii4ZhGMbJkyeNoqIiY+bMmUZlZaVhGIaxYMECY9++fX25xKjW1NRkzJw501i6dKnx+uuvG4Zh\ndPg6bGpqMiZMmGA0NDQYLS0txj333GPU1dWF/bxx0bI+cOCACgoKJEnXXnut6uvrdfbs2T5eVWy4\n+eab9atf/UqSlJaWppaWFpWXl2v8+PGSpLy8PB04cKAvlxgzPv74Y3300Ue66667JIk6huHAgQMa\nM2aMUlNTlZmZqeeff546hiEjI0NnzpyRJDU0NCg9PV3Hjx8PdA6pY9dsNps2btyozMzMwGMdvQ4r\nKyuVlZUlp9Mpu92u0aNHq6KiIuznjYtA9nq9ysjICHztcrlUU1PThyuKHcnJyXI4HJKk0tJS3XHH\nHWppaQm0BN1uN7UM0sqVK7V48eLA19QxdJ9++qlaW1v1+OOPq7i4WAcOHKCOYbjnnnv02WefqbCw\nUDNnztSiRYuUlpYW+D517JrVapXdbr/ksY5eh16vVy6XK3BMT7Mnbt5Dbs9gGmjIdu/erdLSUm3e\nvFkTJkwIPE4tg/P73/9e2dnZuvrqqzv8PnUM3pkzZ7Ru3Tp99tlneuihhy6pHXUMzh/+8AcNGTJE\nr776qv7v//5P8+bNk9P5xefwUsee6ax+Pa1rXARyZmamvF5v4Ovq6mp5PJ4+XFFs2b9/vzZs2KBN\nmzbJ6XTK4XCotbVVdrtdp06duqRtg47t27dPx44d0759+3Ty5EnZbDbqGAa3262cnBxZrVYNHTpU\nAwYMUHJyMnUMUUVFhcaOHStJuuGGG+Tz+dTW1hb4PnUMXUf/njvKnuzs7LCfIy5a1rfffrvKysok\nSUeOHFFmZqZSU1P7eFWxobGxUatWrdLLL7+s9PR0SdJtt90WqOeuXbs0bty4vlxiTPjlL3+pN998\nU7/73e80bdo0zZ07lzqGYezYsTp48KAuXLiguro6NTc3U8cwfPWrX1VlZaUk6fjx4xowYICuvfZa\nvf/++5KoYzg6eh2OGjVKVVVVamhoUFNTkyoqKpSbmxv2c8TNpz29+OKLev/992WxWPTjH/9YN9xw\nQ18vKSaUlJTo17/+ta655prAYy+88IKWLl0qn8+nIUOGaMWKFUpJSenDVcaWX//617ryyis1duxY\nPfXUU9QxRNu3b1dpaakk6V/+5V+UlZVFHUPU1NSkJUuWqLa2Vm1tbXriiSfk8Xi0bNkyXbhwQaNG\njdLTTz/d18uMWocPH9bKlSt1/PhxWa1WXXHFFXrxxRe1ePHiL70O3333Xb366quyWCyaOXOmvvWt\nb4X9vHETyAAAxLK4aFkDABDrCGQAAKIAgQwAQBQgkAEAiAIEMgAAUYBABgAgChDIAABEAQIZAIAo\n8P8Dqjo/39aMVtQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f7c6f082ba8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "Qwn29SjK-XCg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Scikit-learn implementation"
      ]
    },
    {
      "metadata": {
        "id": "-kSEp8MY-y9C",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Note**: The `LinearRegression` class in Scikit-learn uses the normal equation to solve the fit. However, we are going to use Scikit-learn's `SGDRegressor` class which uses stochastic gradient descent. We want to use this optimization approach because we will be using this for the models in subsequent lessons."
      ]
    },
    {
      "metadata": {
        "id": "uKmBKodpgHEE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Import packages\n",
        "from sklearn.linear_model.stochastic_gradient import SGDRegressor\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WuUQwD72NVAE",
        "colab_type": "code",
        "outputId": "c46cd73c-7ca4-4d57-ee4b-0fc636f5cde5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Create data splits\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    df[\"X\"].values.reshape(-1, 1), df[\"y\"], test_size=args.test_size, \n",
        "    random_state=args.seed)\n",
        "print (\"X_train:\", X_train.shape)\n",
        "print (\"y_train:\", y_train.shape)\n",
        "print (\"X_test:\", X_test.shape)\n",
        "print (\"y_test:\", y_test.shape)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_train: (75, 1)\n",
            "y_train: (75,)\n",
            "X_test: (25, 1)\n",
            "y_test: (25,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "MJVs6JF7trja",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We need to standardize our data (zero mean and unit variance) in order to properly use SGD and optimize quickly."
      ]
    },
    {
      "metadata": {
        "id": "VlOYPD5GRjRC",
        "colab_type": "code",
        "outputId": "9d63c0d3-da44-487c-fb8c-c9879f2d97d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Standardize the data (mean=0, std=1) using training data\n",
        "X_scaler = StandardScaler().fit(X_train)\n",
        "y_scaler = StandardScaler().fit(y_train.values.reshape(-1,1))\n",
        "\n",
        "# Apply scaler on training and test data\n",
        "standardized_X_train = X_scaler.transform(X_train)\n",
        "standardized_y_train = y_scaler.transform(y_train.values.reshape(-1,1)).ravel()\n",
        "standardized_X_test = X_scaler.transform(X_test)\n",
        "standardized_y_test = y_scaler.transform(y_test.values.reshape(-1,1)).ravel()\n",
        "\n",
        "\n",
        "# Check\n",
        "print (\"mean:\", np.mean(standardized_X_train, axis=0), \n",
        "       np.mean(standardized_y_train, axis=0)) # mean should be ~0\n",
        "print (\"std:\", np.std(standardized_X_train, axis=0), \n",
        "       np.std(standardized_y_train, axis=0))   # std should be 1"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean: [8.22952817e-17] -1.5617137213060536e-16\n",
            "std: [1.] 0.9999999999999999\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CiE3oLCkOCEa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize the model\n",
        "lm = SGDRegressor(loss=\"squared_loss\", penalty=\"none\", max_iter=args.num_epochs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sGH_pQaDOb49",
        "colab_type": "code",
        "outputId": "ddfff892-314a-4d19-d8a3-549b5e6c555a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "lm.fit(X=standardized_X_train, y=standardized_y_train)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGDRegressor(alpha=0.0001, average=False, epsilon=0.1, eta0=0.01,\n",
              "       fit_intercept=True, l1_ratio=0.15, learning_rate='invscaling',\n",
              "       loss='squared_loss', max_iter=100, n_iter=None, penalty='none',\n",
              "       power_t=0.25, random_state=None, shuffle=True, tol=None, verbose=0,\n",
              "       warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "fA8VkVVGPkTr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Predictions (unstandardize them)\n",
        "pred_train = (lm.predict(standardized_X_train) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\n",
        "pred_test = (lm.predict(standardized_X_test) * np.sqrt(y_scaler.var_)) + y_scaler.mean_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "T8Ws-khqJuNr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Evaluation"
      ]
    },
    {
      "metadata": {
        "id": "Y2pha3VRWd2D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are several evaluation techniques to see how well our model performed."
      ]
    },
    {
      "metadata": {
        "id": "abGgfBbLVjJ_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RKm8IiP7O66e",
        "colab_type": "code",
        "outputId": "4fd58928-36da-4d96-9222-9658e1a0bd58",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((y_test - pred_test) ** 2)\n",
        "print (\"train_MSE: {0:.2f}, test_MSE: {1:.2f}\".format(train_mse, test_mse))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 0.00, test_MSE: 0.00\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TegkJM2-YKEq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Besides MSE, when we only have one feature, we can visually inspect the model."
      ]
    },
    {
      "metadata": {
        "id": "gH5N-U7YQVgn",
        "colab_type": "code",
        "outputId": "4a1ce429-18c5-45e8-e701-2cc7affcb94b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "cell_type": "code",
      "source": [
        "# Figure size\n",
        "plt.figure(figsize=(15,5))\n",
        "\n",
        "# Plot train data\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plt.scatter(X_train, y_train, label=\"y_train\")\n",
        "plt.plot(X_train, pred_train, color=\"red\", linewidth=1, linestyle=\"-\", label=\"lm\")\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Plot test data\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plt.scatter(X_test, y_test, label=\"y_test\")\n",
        "plt.plot(X_test, pred_test, color=\"red\", linewidth=1, linestyle=\"-\", label=\"lm\")\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Show plots\n",
        "plt.show()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2sAAAE+CAYAAAATaYj9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl8lOW9///3ZLIMIROSSSYgoNWq\nLAVkERSxiMQoy7cUVBaNG0JdKi6IiqigWKlr0WqloiiLUioaK8ZTFUU8rW2VFsOhoGLE/qqIQCZ7\nQpLJMOT3B8zNzGSykG0mc7+ej8d5PDJz3zNcXIZz9XMt79tSV1dXJwAAAABARIkJdwMAAAAAAPVR\nrAEAAABABKJYAwAAAIAIRLEGAAAAABGIYg0AAAAAIhDFGgAAAABEoNhwNwDozB544AFt2bJFkrRn\nzx5lZGQoISFBkpSTk6OkpKRmfc/48eO1du1apaent1tbAQDoaG01Tvq89tprmj59epu3E4hUFp6z\nBrSNzMxMPf744xo+fHi4mwIAQMRp7Tjp8Xh07rnn6p///GcbtwyIXGyDBNrJVVddpaeeekoTJkxQ\nXl6eCgsLNXv2bI0fP16ZmZlatWqVcW/fvn21f/9+bdmyRTNmzNDSpUs1YcIEZWZmMigBAKLSDz/8\noOuvv17jxo3TuHHj9PHHH0uSDh06pHvuuUfjx49XVlaWbr31Vh08eFAzZ85UeXm5xo8frx9++CHM\nrQc6BsUa0I527typP//5zxo2bJiee+459e7dW++9957WrFmjpUuXat++ffU+88UXX2jw4MF69913\nlZ2dreeeey4MLQcAoH3Nnz9fZ5xxhjZu3Kjly5frzjvvVFlZmf73f/9XBQUFevfdd/XBBx/o5JNP\n1vbt2/Xwww8rLi5O7733nnr27Bnu5gMdgmINaEdjxoxRTMyRf2YLFy7UokWLJEknnniinE6nvv/+\n+3qf6dq1q7KysiRJAwYMYPYQABB1KioqtHXrVs2cOVOSdMopp2jIkCH661//KofDoa+++koffvih\nqqurNW/ePI0aNSq8DQbChIARoB1169bN+HnHjh3GalpMTIxcLpcOHz5c7zN2u934OSYmJuQ9AAB0\nZhUVFaqrq9PUqVON96qqqnTeeedp0qRJuvfee7V69WrddddduuCCC/TAAw+EsbVA+FCsAR3krrvu\n0jXXXKPLL79cFotFo0ePDneTAAAIi/T0dMXExGjDhg2y2Wz1rk+cOFETJ05USUmJ7rnnHq1atUqT\nJ08OQ0uB8GIbJNBBioqKNHDgQFksFr355puqrq5WVVVVuJsFAECHi4+P1+jRo/Xqq69KOrKqds89\n9+jAgQN6/fXX9fzzz0uSUlNTdcopp8hisSg2NlZer5exE6ZCsQZ0kNtuu01z5szRpEmTVFVVpRkz\nZmjRokX67rvvwt00AAA63EMPPaR//OMfGj9+vC655BKdfPLJ6t69u7KysrRt2zZddNFFmjBhgr79\n9ltdc8016tGjh8444wyNGTNG//73v8PdfKBD8Jw1AAAAAIhArKwBAAAAQASiWAMAAACACESxBgAA\nAAARiGINAAAAACIQxRoAAAAARKCwPhTb5apo9XekpiaqpITnbQSjX0KjX0KjX0KjX0Jrab84nfZ2\naE30askYye9s4+ifhtE3DaNvGkf/NKy5fdPY+NjpV9ZiY63hbkJEol9Co19Co19Co19Co18iF/9t\nGkf/NIy+aRh90zj6p2Ft0TedvlgDAAAAgGhEsQYAAAAAEYhiDQAAAAAiEMUaAAAAAEQgijUAAAAA\niEAUawAAAAAQgSjWAAANcnu8KiipktvjDXdTAACIKB0xRob1odgAgMjkPXxY6zfv1rZ8l4rL3XIk\nJ2hoH6dmZJ4W7qYBABBWjY2R1pi2XQujWAMA1LN+825t2vq98bqo3G28vu3yM8PVLAAAwq6xMTI7\nq0+b/llsgwQABHB7vNqW75IkWeoOK97jNq5tyy9UTe2hcDUNAICw8h8jEzxuqa7OuLYtv7DNt0RS\nrAEAApRVulVc7lafffnKfeoSPfz6QuNaSUWNSsrdjXwaAIDo5RsjF//pV8r53QylVRYZ10oqalRW\n2bZjJNsgAcDk3B6vyird6paUoIQ4q7olJWjBpmc06t+b5bbGa8GMh417U+02pSYnqKKsOowtBgCg\nYwSPkY6yAuU+OUWS9NS4W1VkTzfuTbXb1C0poU3/fIo1ADCpUAekR3W36vpfXqTekl44f7beHjYp\n4DND+6TLFh+rivA0GQCADhFqjJz91bsa/cdnJEnZv3xZFV2SAz4ztE+6EuKsbdoOijUAMKngA9Jn\n/u1tXb/p95Kkgm1fqPrLKqXlF6qkokapdpuG9kknDRIAYAr+Y2TcoVqtXnxkNW3n2ClK/+Nqnb15\nt7Z1wBhJsQYAJuR/QDr2kEdvPDNdMarTP04bqReuXKwlGScou5dVl445NWD7BwAA0c5/jDzju3/r\n1zn3S5LmXrFUpacP0BJvnbKz+nTIGEmxBgAm5Dsg/bNt/6MbPnpRknT39F/ri94DFHP0gHRGaqIS\n4qzKSE0Mc2sBAOg4vjHy7aNn0yoTEnXFL1/R4Rhrh4+RFGsAYAKhQkR8B6QladrNf1RNfBdJ7XNA\nGgCAzsL55TZjjPx374G6b/oS41pHj5EUawAQxUIdkM7qUqYr775MkrSvW3ddP/v5gM+0xwFpAAAi\nUfBkZtpPTlVM4ZEtkDfOXKa9jl4B93f0GEmxBgBRLDhE5L7nbtHpB3ZLkgrf3azXSpIJEQEAmE7w\nZGav+EN67tGpxvX9+0vVf/Nu1YR5jKRYA4Ao5X9AOsFTo5zfXWZcm7n4PS05Y6iy4wgRaanq6mot\nWLBARUVFcrvduummm7Rx40Z9/vnnSklJkSTNnj1b559/vnJzc7VmzRrFxMRo+vTpmjZtWphbDwDm\n5j+ZecXf/6DLtrwuSdp03UIN/vV8WaUOCxFpDMUaAEQp3wHp29/9rTK//F9J0svnXqHXz55GiEgb\n+OijjzRw4EBdd9112rt3r2bNmqWhQ4dq3rx5Gjt2rHFfVVWVli1bppycHMXFxWnq1Km68MILjYIO\nANCx/Ccz3/Y7v33xba+rW6pd/TxeozAL9xjZZLHGzCEAdA5NhYhc/stXVNnFLokQkbYwceJE4+d9\n+/ape/fuIe/bvn27Bg0aJLv9SN8PGzZMeXl5yszM7JB2AgAClVW69ZN/faj5f/6N8d6keRskSSV+\nk5mRoMlijZlDAIhsoUJEJnm+1aW/vtG4xzcI+RAi0nYuu+wy7d+/X8uXL9fq1au1du1arVq1Smlp\naVq0aJEKCwvlcDiM+x0Oh1wuVxhbDADmNqBvDw04+vNdlz2qXT37GdcibTKzyWKNmUMAiGzBISKr\nF08wfi5Z/Qe9HN+XEJF29Oqrr+rLL7/UXXfdpXvvvVcpKSnq37+/XnjhBT377LMaOnRowP11dXXN\n+t7U1ETFxh5/Qe102o/7M2ZC/zSMvmkYfdO4TtM/BQWSXy0TPJEpSecO7qnePdtusam1fdPsM2vM\nHAJA5KmoqtXWXQWSpC7uKr22LNu4NnPxe1py4dmEiLSTnTt3Ki0tTSeccIL69+8vr9erPn36KC0t\nTZKUmZmpxYsXa9y4cSosLDQ+V1BQoCFDhjT5/SUlVcfdJqfTLper4rg/Zxb0T8Pom4bRN43rLP2T\nMmmc4rZ8Ikmqvma2yh5bqqzNu7UtaDJz0jkntdnfp7l901hB1+xirT1mDls6axis01TzHYx+CY1+\nCY1+CS1S+8XrPayVb3+uv2//QaWVtVq2+madVHxkdS3vR0P0wKWLFVNRI2t8nJzpXSVJvdvwz4/U\nfulIW7du1d69e3XfffepsLBQVVVVuv/++7VgwQKdeOKJ2rJli04//XQNHjxYCxcuVHl5uaxWq/Ly\n8nTvvfeGu/kAYBrOjGTjZ9e3B6QuXSIm7bEpTRZr7Tlz2JJZw2CdpZrvaPRLaPRLaPRLaJHYL74Q\nkY3//E4fbftBUmCS1VU3rFJp11RJR/bde2s9bf53aGm/RFuBd9lll+m+++5Tdna2ampqdP/99ysx\nMVFz585Vly5dlJiYqEceeUQ2m0133HGHZs+eLYvFojlz5hhHBgAA7SfhrT8p+bqZxmtXQXn9eyI8\nEbnJYo2ZQwAIv+AQEYtFmvh/7+iXm18w7iFEpGPZbDYtXbq03vtvvPFGvffGjx+v8ePHd0SzAAAK\nXE0reft9HTp7ZBhb03JNFmvMHAJA+AWHiOQuPbaatvqnV+uNsy4xXqckxWt4vwxCRAAApmMpLFT6\nT35svA61mtaZNFmsMXMIAOHlHyLSrapUa5fPNK4Fr6alJiVo8awRsifGd2QTAQAIu26TJyj+k79L\nkqqvnqXK3/w2zC1qvWYHjAAAOpZv6+Nnu1wqrawNOJsmhY4cPrOfk0INAGA6oUJEogHFGgBEmKZC\nRK6/9jntSz1BkhRjkerqJEcyz08DAJhPfO6b6vaLa4zXnX3bYzCKNQCIEKFCRK7f/IIm/d87xj3B\nq2ljhvbSuBEnRmzkMAAAbcU3mekb8wJCRHI36tDIc8LYuvZBsQYAEaKxEJG/9v2pnvh/dxqv/UNE\nrDExHdpOAAA6UvBk5o+s1frdE5cb16NtNc0fxRoARAC3x6tt+S5JUmplsV5+YZZxjRARAIBZuT1e\nrd34lf6+c78k6eHX7tOg7z+XJO3IvFg9Xl0Tzua1O4o1AIgAZZVuFZe7CREBAEDHVtPyvipQcUWt\npMDz25fesl72tG5a4vFG9TEAijUAiADdkhKU6zcIzbn6GX2XfpIkQkQAAObjfzRg8me5+sVfVhrX\nfBOZJRU1Kqt0KyM1MSxt7AgUawAQZl3vmy/niuXGa0JEAABm5n80wH817dc/X6BPTxtpvE6129Qt\nKaHD29eRKNYAIIz8k6zcPz1Py296Umn5hSqpqFGq/dhKGiEiAACzKKt0S3t/0NsrZhvvhToWMLRP\netRPYlKsAUAH8Y8cthUXKn3Q6cY1X5JVtqRLx5waEE0MAEC0Co7jl6QBfXto9dHr32T8WHOvfDLg\nM2nJCRrax2mKYwEUawDQzoIjh3ODQkSCI4cT4qxRvf8eAIDgsdFxtAC7LXu4cc/0m/+o6vguAZ8b\nNbCHrhrX1zSTmeyrAYB25jskXRRUqK197NWofjYMAAAN8R8b6ySd+7+vBxRq+/eX6txRpyst2aYY\ni5SWbFPW8N66dmI/0xRqEitrANCufIek573zlMbu+ovx/qR5G5RWbdP5UR45DABAMP8AESkwROSZ\nGffrkidvV0JMjLKz+pj+aADFGgC0o7JKt1YvnmC89t97b4bIYQAAgvmeLdq9dL9eXHmj8f6keRsU\nY5Eu8BsbzX40gGINANqQ/0HpLoUHNGBwP+NacJKVGSKHAQAIFvxsUenYGMnYGIhiDQDaQFMhImaN\nHAYAIFjvXqnGz5fdtFYHbUnGa8bGQASMAEAruD1eFZRUad0H+SFDRP645GXt31+qrOG96x2SNkPk\nMAAAPl0XLwx4vujT67bKlpHO2NgIVtYAoAWCV9IsFumBP/1Kw/+bZ9wzad4GpR2yabS3jkPSAABT\nCX5+mn+RVvGbp1Vz9bU8W7QZKNYAoAV8kcM+uUuPraaV2+y64qZXJAWGiJj9kDQAIPoFT2b28Zbo\nN09fa1zn2aLHh2INAI6Tf+RwRtkBvfTSDcY1QkQAAGbmP5n5dtD5bZ4tevwo1gDgOPkih4MHIUJE\nAABm5fZ45SqpMiYz/cfIy25aK1tGupbwbNHjRrEGAMcpOHL43qkPacdJgyRJMRaprk5yJNs0tE86\nB6UBAFHN6z2sdZvytS3fpaJyt+a985TG7vqLcd03kVnNs0VbhGINAI5DyoRMxX221XgdvJo2Zmgv\njRtxIgelAQCmsPLtz0Nue3zl3Cv02tnTjNccC2gZijUAaCb/JCvpSORwWn6hSipqlGo/tpJmjeGp\nKACA6Of2ePXpzn06qfBbLXv5NuN9jgW0HYo1AAjB7fFqX+FBeT1eddm/V2lnDjSu+Q5IEzlsbtXV\n1VqwYIGKiorkdrt10003qV+/fpo/f768Xq+cTqeeeOIJxcfHKzc3V2vWrFFMTIymT5+uadOmNf0H\nAECEK6t066VF4wLe8y/ULBbJYedYQGs0WawxGAEwk4DI4Qp3QCS/ROQwjvnoo480cOBAXXfdddq7\nd69mzZqlYcOGKTs7WxMmTNCTTz6pnJwcTZkyRcuWLVNOTo7i4uI0depUXXjhhUpJSQn3XwEAWmVA\n3x7Gz1fdsEqlXVON1w57guZOHyxnShcmM1uhyb06vsFo7dq1+u1vf6tHH31UzzzzjLKzs7Vu3Tr9\n6Ec/Uk5OjqqqqrRs2TKtXr1ar7zyitasWaPS0tKO+DsAQKu5PV4VlFRp3aavtWnr9yoqDyzU/nTv\n74kcRoCJEyfquuuukyTt27dP3bt315YtW3TBBRdIksaOHatPPvlE27dv16BBg2S322Wz2TRs2DDl\n5eU19tUAENG6XTop4GjApHkbAgo1SRrW16neziQKtVZqcmVt4sSJxs/+g9GDDz4o6chgtHLlSp1y\nyinGYCTJGIwyMzPbqekA0Hr+K2lF5W7FWKTlK29Sr9IfjHsmzdugtHibziJyGCFcdtll2r9/v5Yv\nX65rr71W8fHxkqS0tDS5XC4VFhbK4XAY9zscDrlcrnA1FwBaxb9Iq/rFDUpY/ntlvbZN20Kc4Ubr\nNfvMGoMRgGjk//BOSXpraehnp5UQOYwGvPrqq/ryyy911113qa6uznjf/2d/Db0fLDU1UbGxxz85\n4HTaj/szZkL/NIy+aRh9I+nzz6WBx85vq65OvhHxtsvPVE3tIZWUu5WanCBbPLEYPq393Wl2T7bH\nYNTSgSgY/4BCo19Co19CM2O/lFW6lXf04Z0nlOzTC6t+aVwLTrJKT+miU09OYwA6yoy/L8F27typ\ntLQ0nXDCCerfv7+8Xq+6du2qmpoa2Ww2HThwQBkZGcrIyFBhYaHxuYKCAg0ZMqTJ7y8pqTruNjmd\ndrlcFcf9ObOgfxpG3zSMvqmfhuwqKJeO9ol//8RKqiirlrl765jm/u40NqY2+b862nMwaslAFIx/\nQKHRL6HRL6GZrV98Wx8/2+VSSaU74LkwUujI4TNOTWMAOqqlvy/RVuBt3bpVe/fu1X333afCwkJV\nVVVp9OjR2rhxoyZPnqz3339fo0eP1uDBg7Vw4UKVl5fLarUqLy9P9957b7ibDwDN4l+oFe3I1+Hu\nPRq5G22tyYCRrVu3auXKlZJkDEajRo3Sxo0bJSlgMNqxY4fKy8t18OBB5eXlafjw4e3begA4DsEh\nIsGF2sOT7jYKtRjLkf9LS7Ypa3hv9t6jnssuu0zFxcXKzs7W9ddfr/vvv1+33HKLNmzYoOzsbJWW\nlmrKlCmy2Wy64447NHv2bF177bWaM2eOcb4bACJVt0t/HlCouQrKKdTCwFLXxH7Fmpoa3Xfffdq3\nb59qamp08803a+DAgbr77rvldrvVs2dPPfLII4qLi9N7772nl156SRaLRVdeeaV+/vOfN/qHt8VM\nvtlWBJqLfgmNfgkt2vslVIhIQ2fTfMYO7anLx/9E3loPoSJBWFnrGC3t42j+t9xa9E/D6JuGmbFv\n/Iu06lnXqfLRpQ3fa8L+aa4O2QZps9m0dGn9/0CrVq2q99748eM1fvz4JhsEAB2puSEikpSSFK/h\n/TI0I/M09UjvygAEADAN61e75Bh9lvGaR9aEHyflAUS1iqpabd1VIEnqXbRHz625xbgWvJqWmpSg\nxbNGyJ4Y36FtBAAg3EKGiCDsKNYARCX/EJHSytpmhYic2c9JoQYAiHpuj1dllW51S0pQQpyVEJEI\nRrEGICr5b330L9R+O+4WfTjgAklHAkTqJDl4gCcAwAT8z3AXl7v18IYHNfA/24zrrKZFHoo1AFHH\n7fFqW76rydW0MUN6atxZJxkziwAARLOGJjK3XzhNPf/wUriahUZQrAGIOmWVbq1ePCHgvYZCRKwx\nTT7BBACATs83kdmreK+Wr55jvD9p3galJdu0xONl4jICUawB6PT8994nfpOvAeedbVwjRAQAgMYn\nMksqalRW6VZGamI4moZGUKwB6LSC997nEiICAEBIA/oeCw256oZVKu2aarxOtdvULSkhHM1CE9j/\nA6DTcXu8Kiip0roP8rVp6/cqCirUNl+7QPv3lypreG+lJdsUY5HSkm3KGt6bEBEAgKkkz746IO1x\n0rwNAYWaJA3tk84WyAjFyhqATiN4Jc1iUcgQkbRkm5Z465Sd1UeXjjk1IJ4YAACz8C/Sqm66VeX3\n/0pZm3drW36hSipqlEoacsSjWAPQafinWElS7tLQ2x79994nxFnZgw8AMBXrN1/Lcc6ZxmtfJL9V\nYiKzk6FYA9ApVFTV6rNdLklS76I9em7NLca1eiEi7L0HAJiU/2qaFPrZaUxkdh4UawAimm/r49Zd\nBSqtrG3y2WkSe+8BANHPPwnZN+b5F2pFO/J1uHuPhj6OToJiDUBEa+gBnssuuFHvDR4vSYqxSHV1\nkiOZvfcAgOgWfH7bkZyge997Un0+/cC4J9RqGjonijUAEcv3AM+mVtPGDO2lcSNOZO89ACDqBZ/f\n9n92WtWNN+vgrx4OR7PQTijWAESsxh7gKR15wPWZ/ZyakXmarDE8iQQAEN18k5iSdELJD3ph1U3G\ntZmL39OS684WJ7ajC8UagIjhv/8+8Zt8DTjvbONa8GpaSlK8Fs8awQOuAQCmUVbpVnG5O+SOkxi/\nJGRED4o1AGEXvP8+txkhIsP7ZVCoAQBMpVtSQsAYec31K1Wc5JBEEnK0Yt8QgLBxe7wqKKnSug/y\ntWnr9yoKKtT+euXt2r+/VFnDeyst2aYYi5SWbFPW8N6EiAAATMV+4yz17pVqvJ40b4NRqEkkIUcr\nVtYAdLjglTSLRSG3dKQl27TEW8cDPAEApuYfyX/whpv04phZSssvVElFjVLtJCFHM4o1AB0uOMkq\nd2nobY8lfvvveYAnAMAMAs5v7/n/5Bg5zLjmi+TPlpjENAmKNQAdyj/JqnfRHj235hbjWvDZNPbf\nAwDMoqnz28HPTmMS0xwo1gB0qMaSrIKx/x4AYBb+u078x8gXl72rydPODVezEGYUawDalf92joQ4\na70kq5XnzdSbw4+8jrFIdXWSI5n99wAA8/DtOrn2L6t1yWfHJi8nzdugNJdF4z1eJi9NimINQLsI\n3s7hSE5o9AHXkjRmaC+NG3Ei++8BAKZSVukOGCM3DPu5Xjp/lqTA89swH4o1AO0iOEQkuFB7et3W\nkElW1hieKAIAMI+Y7/dowLABxmvOb8MfxRqANldRVavPdh0JEelVvFfLV88xrs1c/J6WXHe2suOs\nJFmh03v88cf12Wef6dChQ7rhhhu0efNmff7550pJSZEkzZ49W+eff75yc3O1Zs0axcTEaPr06Zo2\nbVqYWw4gEvhH8kuc30Z9zSrWGIwANIdv6+PWXQUqrawNGSISQxw/osSnn36qr7/+WuvXr1dJSYku\nvvhijRw5UvPmzdPYsWON+6qqqrRs2TLl5OQoLi5OU6dO1YUXXmiMoQDMyb9QK9y5W4fS05W1ebe2\n8fw0+GmyWGMwAtAUX4jIxn9+p4+2/SApMMlq5ehr9OaIiyWxnQPRY8SIETrjjDMkScnJyaqurpbX\n66133/bt2zVo0CDZ7XZJ0rBhw5SXl6fMzMwObS+AyND1wUVKXPa08doXyW+VlJ3Vh10nCNBkscZg\nBKAhwSEiFouajORnOweihdVqVWLikZXhnJwcnXfeebJarVq7dq1WrVqltLQ0LVq0SIWFhXI4HMbn\nHA6HXC5XuJoNIIz8V9MO3rNIVbffVe8edp3AX5PFWnsORqmpiYqNbf3/aHM67a3+jmhEv4RGv4TW\nkn5ZsWFHQIhI7tKGC7W0bjade0ZPzZo0QFZr5wkR4fclNPrlmE2bNiknJ0crV67Uzp07lZKSov79\n++uFF17Qs88+q6FDhwbcX1dX16zvbekYyX+bxtE/DaNvGtbqvtmzRzrppGOv6+rUVVLX1n1rxOB3\np2Gt7ZtmB4y0x2BUUlJ1/C0O4nTa5XJVtPp7og39Ehr9ElpL+qWiqlYfb9srSepd/L2eW32zcW3S\n7W9KFovxOiUpXvdfM1z2xHgVFx9sm0Z3AH5fQmtpv0TjYP7xxx9r+fLlevHFF2W323XOOecY1zIz\nM7V48WKNGzdOhYWFxvsFBQUaMmRIk9/dkjGS39nG0T8No28a1tq+CQ4RcRWUS1HU1/zuNKy5fdPY\n+Nis6W3fYLRixQpjMOrfv7+kI4NRfn6+MjIy6g1GGRkZzfl6AJ2I9/BhrduUrwdW/lMllW69/eSU\nwEJt3oaAQk2ShvfLkD0xvqObCrSriooKPf7443r++eeN89m33HKL9uzZI0nasmWLTj/9dA0ePFg7\nduxQeXm5Dh48qLy8PA0fPjycTQfQQYJDRHzn04DmanJlzTcYrV69OmAwmj9/vk488cSAwWjhwoUq\nLy+X1WpVXl6e7r333nb/CwDoWP7PT/M/n/aHcy7Xq+fMkCTFWKS6OsmRTJIVotc777yjkpISzZ07\n13jvkksu0dy5c9WlSxclJibqkUcekc1m0x133KHZs2fLYrFozpw5xvluANHDF7bVLSlBjocXhwwR\nAY5Xk8UagxEAH7fHq235riZDRMYM7aVxI04kyQpRbcaMGZoxY0a99y+++OJ6740fP17jx4/viGYB\n6GDBYVu5fmPkwbvvU9Udd4exdejsmizWGIwA+JRVurV68YSA9/wLtdSkBJ3Zz6kZmafJGtN5QkQA\nAGgp346T9AqXcldcZ7z/9Lqtys7qE8aWIRo0O2AEgPn4b+lI/M/XGjD6LONaqBCRxbNGcDYNAGAa\nje04Scsv1KVjTmWHCVqFYg1APY1t6ZDqb3uUCBEBAJhP8I6Tq25YpdKuqZKkkooalVW6eWYaWoV9\nSgDq8W3pKAoq1LZOulr795cqa3hvpSXbFGOR0pJtyhremxARAEDUc3u8KiipktvjVdeHHtCAvj2M\na5PmbTAKNUlKtdvULSkhHM0aj2uXAAAgAElEQVREFGFlDUCARrd0JNu0xFun7Kw+unTMqcYWSbZ4\nAACiWWM7Tj6ZeoMePmlCvc8M7ZPO+IhWY2UNQIDGQkR8WzokKSHOqozURAYiAEDUMx5bs3dvQKH2\n9LqtOuXZx9hxgnbDyhpgcm6PV/sKD8rr8Srxu/9owDlnGteCQ0TY0gEAMJua2kNNhoiw4wTthWIN\nMKmALR0VbuUubTpEhC0dAACzKSkP3HFyxY1rVJ7Y7cg1vxAR344ToC2xDRIwqYAQEb9CbcuU2YSI\nAABMyz9EpMuK53SCM8m4NmneBqNQk9hxgvbHyhpgQoSIAAAQqLEQkc2zFuiplJH1PsOOE7Q3ijXA\nhJoTIsKWDgCAmfh2nKQcLFXu8zON91e8+W9NOuckZW3erW35hSqpqFGq3aahfdLZcYJ2R7EGmIDb\n4zVWyBK//Y8GjCJEBAAAn1A7Tg7Losnz3lTGzn2acNaJ7DhBWFCsAVGssS0dEiEiAABI9Xec+IeI\nFJZWs+MEYUPACBCFfIej1236+liIiF+htm3C5YSIAAAgyfbS8xrQt4fxOjhEJD2lCztOEDasrAFR\nxH8lrajcrRiLmhUiYo2Pk7fWw4oaAMBUnBnJxs+br12gp1Lrh4iMHHgC4yPChmINiCK+w9E+bzXw\n7LTgEBFnele5XBUd2lYAADqK/9nthDirLC6X0gecalx3FZTrJ4cPhwwRmTVpgIqLD4ax9TAzijUg\nSvgOR0tS99L9enHljcY1QkQAAGYUfHbbkZxQLw3ZVVAuSbLGxIQMEbFaOTWE8KFYA6JEWaVbxeXu\nkNsegxEiAgAwg+AdJ/6FWuEX/1Fdenq9zxAigkjCVAHQifmCRNwer7olJQSEiPxx5AyjUIuxHFlY\nI0QEAGAW/jtOJv7fOwGTmTMXv6eabqnhahrQbKysAZ3Q8UbyjxnSU+POOonnwgAATCPUjpPnMq/X\nO0MmKsbv7DYQySjWgE7If1tH8LbHp9dtVVrQ4egZmafJGsNCOgDAPFIOVQdMZvpPZHJ2G50FxRrQ\nibg9XrlKqrQt36XuZQf04ks3GNcm3f6m0rp10ZIxp9Y7HA0AgJmkjB+ruLzPJElf9Thdd2Y/EXCd\ns9voLCjWgE4g+PlpDYWI+Efys7UDAGBG/s9OO7Drv3ovrzDkjhOgM6BYAyKc2+PVKxu/0j927pcU\nuO3x1bOn6Q/nXmG8ZlsHAMCs4t9+S91mX2W8dhWUK0ZSdpaDHSfotCjWgAjV3NU0f2zrAACYkf9q\nWtlLr6h20uSA68Txo7OiWAMiVGMhIv6FmsUiOdjWAQAwIUtpidL7/Mh47XvANRAtmlWsPf744/rs\ns8906NAh3XDDDRo0aJDmz58vr9crp9OpJ554QvHx8crNzdWaNWsUExOj6dOna9q0ae3dfiDq+IeI\nOMsLtPLF641rk25/80h1dpTDnqC50wfLmdKFFTUgDBgfgfBJmZCpuM+2SpI8w85U6XsfhblFQNtr\nslj79NNP9fXXX2v9+vUqKSnRxRdfrHPOOUfZ2dmaMGGCnnzySeXk5GjKlClatmyZcnJyFBcXp6lT\np+rCCy9USkpKR/w9gE6vJdseh/V1qrczqaOaCMAP4yMQPv7bHgvzv1VdCg+4RnRq8sFLI0aM0NNP\nPy1JSk5OVnV1tbZs2aILLrhAkjR27Fh98skn2r59uwYNGiS73S6bzaZhw4YpLy+vfVsPRBHftsfg\nQu21sy6tV6g57AnKGt6bbY9AGDE+Au3P7fGqoKRKbo9X0pEQEf9CzVVQTqGGqNbkyprValVi4pED\nmTk5OTrvvPP0t7/9TfHx8ZKktLQ0uVwuFRYWyuFwGJ9zOBxyuVzt1Gwgurg9Xm3LdzVrNe3cgT10\n5bi+bHsEwqy9x8fU1ETFxh7/v3On037cnzET+qdhkdQ3Xu9hrXz7c326c59cpdVypnTRS4vGHbvh\n9delqVPl7KD2RFLfRCL6p2Gt7ZtmB4xs2rRJOTk5WrlypS666CLj/bq6upD3N/S+v5YORMH4BQmN\nfgktEvtlX+FBrV48IeC94BARZ0oXjRx4gmZNGiCrtclF8eMWif0SCeiX0OiXY9pjfJSkkpKq426L\n02mXy1Vx3J8zC/qnYZHWN+s25RshW11rKvXS0mOTmUaISAe1N9L6JtLQPw1rbt80NqY2q1j7+OOP\ntXz5cr344ouy2+1KTExUTU2NbDabDhw4oIyMDGVkZKiwsND4TEFBgYYMGdLo97ZkIArGL0ho9Eto\nkdIvbo/XeN5Ll33f64Thg4xrTYWIFBcfbPP2REq/RBr6JbSW9ks0FnjtNT4CZubbbSJJG566RNa6\nw5Kk/O6n6+Ff/k5LPF52l8A0mpyer6io0OOPP67nn3/eOAw9atQobdy4UZL0/vvva/To0Ro8eLB2\n7Nih8vJyHTx4UHl5eRo+fHj7th7oZLyHD2vdpnwtXPGp7nn+U/Xulao0/0Jt3oaAQk06FiLCwARE\nFsZHoH2UVbpVfPT8tq9Qu/yXr+iOK55QSUWNyirdYW4h0HGaXFl75513VFJSorlz5xrvPfroo1q4\ncKHWr1+vnj17asqUKYqLi9Mdd9yh2bNny2KxaM6cObLbo28WFWgpt8erVzZ+pX/s3C8p8Nlp28Zd\nphPWLFfW5t3all+okooapfLsNCCiMT4C7aPnG2uV++Sdxmv/YwGpdpu6JSWEo1lAWFjqmrt5vh20\nxfYitimFRr+EFo5+aU4kf1qyTUuuO1sJcdaALZIdtZrG70to9EtobIPsGC3tY35nG0b/NCxS+sY/\n6fGF83+ht4f9LOB61vDeys7q07FtipC+iVT0T8M67MwagJbzRfJLajDt0betIyM1UQlxVmWkJnZ4\nOwEACBdLUZHS+59ivN6/v1TVm3crjd0mMDmKNaCduD1euUqqtC3fJWe5SytfvM649vPb/6Q6y7Ej\no2zrAACYVXr3brL4bfRyFZTLKik7q48uHXNqh+82ASIJxRrQxpqz7THY0D7pDEIAANPx3/ZYuHO3\n6jIyAq6z2wRm1/YPawJMzO3xatU7u7Rp6/f1CrW3h0ysV6g57AnKGt6bbR0AAFOxrX4poFBzFZTX\nK9QAsLIGtImWrKadO7CHrhzXlxU1AICp+BdplUseVfX1N4WxNUBko1gD2kBzQkSkI49Qc/gdkrbG\nsLgNADCH4BARV0F5GFsDdA4Ua0Ar+IeIpFe4tGpFwyEiDnuC5k4fLGdKF1bTAACmkt7TIcuhQ8Zr\nCjWgeSjWgBZoybbHYX2d6u1M6qgmAgAQEZoKEQHQMPZgAcfJ7fFqdQMhIm8N/RkhIgAASLKtWUmI\nCNBKrKwBzeRbTcv7qkDFFbWEiAAA0ICAEJGHHlH1DXPC2Bqg86JYA5qJEBEAABpnKS5Sej9CRIC2\nQrEGNME/RMRZ7tLKFwkRAQAgGCEiQNujWAMaQIgIAADNExAisuNr1XXvHsbWANGD/VlACG6PV6sa\nCBF5f2BWvUItLZkQEQCA+dheXlU/RIRCDWgzrKwBflqymjZqYA9dRYgIAMBkCBEB2h/FGuCHEBEA\nABpHiAjQcSjWgKPcHu/REJECrXzxeuP9yXPf0OGYY6tmhIgAAMzC7fGqrNKtbkkJSoizKr13uiy1\ntcZ1CjWgfVGsAUeVVbq1evGEgPcIEQEAmJH/sYDicrccyQkBYyQhIkDHoFiDaQXPFg7o28O49nGf\nc/X4z+4KuN9hT9Cwvk5CRAAAUc//WMBFOz7QLR8sM66xmgZ0HIo1mE7wbGFuM0JEzh3YQ1cSIgIA\nMAHfsQAp8Pz2i2Ou1d/GztASj5fxEOggFGswncZCRPbvL1XW5t3all+okooapRIiAgAwmbJKtw7v\nO6C3n59pvOebyIypqFFZpVsZqYlhah1gLhRrMJWa2kMNhoikpnTVEm+dsrP66NIxpwZskQQAIFq5\nPV7tKzwo79EVswF9e+hlv+v+O05S7TZ1S0ro+EYCJkWxBlMpKW84RKTEb7YwIc7KrCEAIKoFHAuo\ncMthDwwRmfWLF+RKzgj4zNA+6UxiAh2IfV0wlRP8Uhy3njyM2UIAgGn5jgUUlbt16ZacgEJt//5S\nDc4cprRkm2IsUlqyTVnDexOyBXQwVtZgCs6M5IDXoUJEmC0EAJiB2+OVq7RaeV8VSAo8v/3mmZP1\n1qRfciwAiBDNKtby8/N10003aebMmbryyiu1YMECff7550pJSZEkzZ49W+eff75yc3O1Zs0axcTE\naPr06Zo2bVq7Nh5ojuBCrbEQEQA4HoyP6EyC05C7HSzR289fa1wPFSLCsQAgvJos1qqqqvTQQw/p\nnHPOCXh/3rx5Gjt2bMB9y5YtU05OjuLi4jR16lRdeOGFxoAFdAT/Z6d1OfCD0oYNMK659pXI2SNF\nVlcFs4UAWo3xEZ1NY2nIHAsAIlOTZ9bi4+O1YsUKZWRkNHrf9u3bNWjQINntdtlsNg0bNkx5eXlt\n1lCgMd7Dh7VuU74WrvhU9zz/qXr3Sg0s1ArKJWtgQeabLaRQA9ASjI/oTBp6dtovZi2vdzSAYwFA\n5GhyZS02NlaxsfVvW7t2rVatWqW0tDQtWrRIhYWFcjgcxnWHwyGXy9W2rQVCcHu8WrvxK/19535J\ngYPQt4NGKvHD98PVNABRjPERnUlZpVvj312ly7a8ZrznX6RZJDmSORYARJoWBYxMnjxZKSkp6t+/\nv1544QU9++yzGjp0aMA9dXV1TX5PamqiYmNbP3PjdNpb/R3RKNr7xes9rJVvf65PdvwgV2lNyC0d\nGaldtKxbF9nij/2qR3u/tBT9Ehr9Ehr9ElpbjY9Sy8dI/ts0zqz948xIlm+/ycaBF+rZi+YY1zJS\nu2jR7JHqkZYYMF7iGLP+3jQX/dOw1vZNi/5F+u/Pz8zM1OLFizVu3DgVFhYa7xcUFGjIkCGNfk9J\nSVVL/vgATqddLldFq78n2pihX9Ztym9y731habW++W+RcTjaDP3SEvRLaPRLaC3tFzMM5m01Pkot\nGyP5nW2cGfvHUlCg9IHHVspCpSGfcWqaTj4hWS5XhczVO81jxt+b40H/NKy5fdPY+Nii56zdcsst\n2rNnjyRpy5YtOv300zV48GDt2LFD5eXlOnjwoPLy8jR8+PCWfD3QKLfHq+8LKrQt3yVHRVFAoTZ5\n7hsckgYQNoyPiCTOjOSAQm3//lJlDe/Ns9OATqTJlbWdO3fqscce0969exUbG6uNGzfqyiuv1Ny5\nc9WlSxclJibqkUcekc1m0x133KHZs2fLYrFozpw5stujfxYVHcc/crio3N1okpUPh6QBtBfGR0Qy\n/8fWFG35Px0+5ceySqQhA52Mpa65m+fbQVssmbL0Glo09ktD2x4/PfUs/XryvQH3piUnaGgfp2Zk\nniZrzLEF5Gjsl7ZAv4RGv4TGNsiO0dI+5ne2YWbon8RHl6jrk48br10F5c36nBn6pqXom8bRPw1r\ni22QnCJFp+CLHG7OatqogT101bi+zBYCAKKa/7NFE+KsAatp1dlXqfK3y8LYOgBtgWINnUJZpVur\nF08IeC8gctgiOezHIof9V9MAAIgm/scCisvdOjmmSs/8Jtu43tzVNACRj2INEcl/trBLUYEGnNHX\nuDZ57hs6HHNs1cxhT9Dc6YPlTOnCahoAIOqt37y7wTRkCjUgulCsIaIEzxbmNmPb47C+TvV2JnVU\nEwEACBvfsQApsFC7btZz8px0ipZ4vExcAlGEvWKIKL7ZwqKgQu0/Q0cTOQwAML2ySrdG/vXNgEJt\n0rwN2p9ygkoqalRW6Q5j6wC0NVbWEDEaCxFJS7ZpibeOyGEAgKkEh4gM6NtDA45eW3/2NK099wrj\nXp4tCkQfijVEjMZCRHyzhRmpiUqIsyojNTEcTQQAoEMEHws4MdatZY/PMK7zbFHAHCjWEDb+s4W2\nkiINGHhsO2NwiAizhQAAM2ksRGT//lJlbd6tbfmFKqmoUapfGjKA6EKxhg7XkhARZgsBAGbRUIjI\nrF+s0OHeJ3IsADARAkbQodwer1a/sytkiMjXZ11AiAgAwPTKKt3qv3VzvRARV7IzIETEdyyAQg2I\nXqysoUP4VtPyvipQcUWt/vD7q5RcU2FcJ0QEAIAj/ENEnsu8Xu8MmWhc41gAYC4Ua+gQje29J0QE\nAADJUl6m9NNONF5zLAAAxRraldvjlau0WnlfFSi5ulx/eO5q49qU23LktR77FWS2EABgVqnnna3Y\nXV9KkuoSEnTg2wOEiACgWEP7IEQEAIDmcWYkGz8X7cjX4e49ZJU4FgCAgBG0D9+2x+AQkY0DL6xX\nqKUlJxAiAgAwnfi3NwQUaq6Cch3u3iPgHkJEAHNjZQ1tzhc5/PxLN6pn2X7j/VCraaMG9tBV4/oy\nCAEAopr/s0UT4qwBRVrFI0+oZvYNYWwdgEhFsYY2V1bp1urFEwLe8y/ULJIcycf23ltjWOAFAESn\n4GMBPeMPafmjU43rroLyMLYOQKSjWEOr+c8W2ipKNaDfKca14BCRtOQE3Tb1DDnZ0gEAMAH/NORn\nXr5NpxR+K0k6FBunkh+Kwtk0AJ0AxRparGUhIk71zrB3VBMBAAgb37EAKfCxNVdfv1IxPXtqicfL\nxCWARrH/DC3WUIjI52Mmaf/+UmUN7620ZJtiLFJaso0QEQCAqZRVunXCF3kBhdqkeRtUkuQwni0K\nAI1hZQ0t4pstfG7VHPUu2Wu8P2neBqUl27TEW0fkMADAVIJDRAb07aFHjl57eNLd+uT0c4x7ebYo\ngOagWEOLNBYi4pst9EUNZ6QmhqOJAAB0iOBjAT1sdXrh4YuN6zxbFEBLUayhWQJCRCrLNKDvyca1\n4BARZgsBAGbiHyJyy8bf6aLPP5Qk7RkwQnEffqCszbu1Lb9QJRU1SrUfS0MGgKZQrKFRLQsRYbYQ\nAGAODYWIXPHLlxXXPYNjAQBahYARNMjt8Wr1O7tChoh8cd7PCBEBAJheWaVbif/f7nohIuVdkgNC\nRHzHAijUAByPZq2s5efn66abbtLMmTN15ZVXat++fZo/f768Xq+cTqeeeOIJxcfHKzc3V2vWrFFM\nTIymT5+uadOmtXf70Q58q2l5XxWouKJWy9bcopOK9hjXCREBAOCI066crN//a4sk6Y7LH1f+CX2M\naxwLANBaTa6sVVVV6aGHHtI55xxLMHrmmWeUnZ2tdevW6Uc/+pFycnJUVVWlZcuWafXq1XrllVe0\nZs0alZaWtmvj0T58e++LK2r19pNT6hVqkpgtBAAdmczMysrS2rVrJUn79u3TVVddpezsbN12222q\nra2VJOXm5urSSy/VtGnT9Prrr4ezyWgrbrecGclKOFqoTZq3IaBQkzgWAKD1mizW4uPjtWLFCmVk\nZBjvbdmyRRdccIEkaezYsfrkk0+0fft2DRo0SHa7XTabTcOGDVNeXl77tRxtzu3x6ntXpfK+KlDX\nmsqALR1TbssJOJ/GbCEAs2My07xsq1+S80SnJOngXfdwLABAu2lyG2RsbKxiYwNvq66uVnx8vCQp\nLS1NLpdLhYWFcjgcxj0Oh0Mul6uNm4v2QIgIABw/32TmihUrjPe2bNmiBx98UNKRycyVK1fqlFNO\nMSYzJRmTmZmZmWFpN1rHmZFs/Fz4n72qS7LLKnEsAEC7aHUaZF1d3XG97y81NVGxsa3/f2ZOp73V\n3xGNmtMvNbWH9Nwb/9bmo5HD/qtpm/ufr6cmzA24PyO1i0YOPEGzJg2Q1do582n4fQmNfgmNfgmN\nfmEy02ys+V/J8dMRkiTvCT1VvH1XvXt4tiiAttaiYi0xMVE1NTWy2Ww6cOCAMjIylJGRocLCQuOe\ngoICDRkypNHvKSmpaskfH8DptMvlqmj190SbpvolOETkd2tu1clF3xnXQ62mjRrYQ1eN66uEOKuK\niw+2S7vbG78vodEvodEvobW0X8xW4LVmMlNq+YSm2fr5eLWof376U+nvfz/y8yefyDpypJxt26yI\nwO9Ow+ibxtE/DWtt37SoWBs1apQ2btyoyZMn6/3339fo0aM1ePBgLVy4UOXl5bJarcrLy9O9997b\nqsah/fg/wPPtRrY9WiQ5ko89wNMa0zlX0wCgI7TVZKbUsglNJhgad9z943YbZ9MkyVVQfvSH6Otj\nfncaRt80jv5pWHP7prGCrslibefOnXrssce0d+9excbGauPGjfrNb36jBQsWaP369erZs6emTJmi\nuLg43XHHHZo9e7YsFovmzJlj7M9H5HB7vHKVVGlbvkuJ7oNav+wK49rFt72uQ9Y443VacoJum3qG\nnCQ9AkCzMJkZPWxrVsp+15GjAAfvXKCq+fw3A9DxmizWBg4cqFdeeaXe+6tWrar33vjx4zV+/Pi2\naRnalH+ISFG5u9HVNJ+hfZzqnUHBDQChMJkZvQJCRL75XnX25EbuBoD20+qAEUQ+t8ertRu/0t93\n7pcUuO3xvUEXatmFcwLuT0tO0NA+TiKHAaARTGZGH+vX+XKcO1yS5O1xgor//VWYWwTA7CjWolhw\niMij6+/RgL1fGtebChEBAMAsUiaNU9yWTyRJJX/+QIdGnB3mFgEAxVpUa3aIiEVy2AkRAQCYUG2t\nnL3TjZdGiAgARACKtSjk9nj1331lyvuqQF3cVXptWbZxLThExGFP0Nzpg+VM6cJqGgDAVGwvr5L9\nztskESICIDJRrEUR/xCR4nK3cpsRIjKsr1O9nUkd1UQAACICISIAOgOKtSjh9nj1ysav9I8QISLv\nD8zS7y66OeB+QkQAAGYUECLSvYeKd+SHuUUA0DCKtU4uOJJ/4YZf6+z//Mu4TogIAABHnXeeHB9/\nLIkQEQCdA8VaJ9fsEBFJjmRCRAAAJkSICIBOimKtk3J7vHKVVGlbvqvJEJG05ATdNvUMOVMTWU0D\nAJiK7ZXVst9x65EXixbJdctd4W0QABwHirVOJnjbY2OraT5D+zjVO8PeUU0EACAiBIeIpP+4l+Sq\nCGOLAOD4UKx1Io2FiHww4AI9M+6WgPsd9gQN60uICADAXKy7v5Zj1JmSJG9GdxXv/DrMLQKAlqFY\n6wSCV9Pue+thjfzmn8b1UKtpmcNP1LQxP2bbIwDAVLpNmaj4f/xNklTyPx/o0FmEiADovCjWOoFm\nh4hYJIf9SIjIzdOHqLj4YIe2EwCAsCFEBEAUoliLYP4hIgmeGuX87jLjWnCIiMOeoLnTB8uZ0kUJ\ncVZZraQ9AgDMwT9E5OC8+apasDDMLQKAtkGxFoFaEiIyrK9TvZ1JHdVEAAAiQkCIyO49qkvuFsbW\nAEDboliLMI2FiLwxfIpWnzcz4H5CRAAAZuQfInLYmaGiz3eHuUUA0PYo1iJE8Gra9ZtXaNL//dm4\nHmo17dyBPXTluL6EiAAATKXbxf9P8X//WBIhIgCiG8VamLk9XpVVurXxX3v0Ud5eSc0PEZmReZqs\nMZxNAwCYBCEiAEyGYi1MglfSYixSvMetN343w7inqRARAADMwrZ2jezzjjxP9OC8u1S1YFGYWwQA\n7Y9iLUz84/gl6a2lhIgAABAKISIAzIpiLQwqqmr12S6X8dp/2+PK0dfozREXB9xPiAgAwIys33wt\nxzlHQ0TSnSr64pswtwgAOhbFWgfybX3cuqtApZW1uuRff9K1H79sXCdEBACAI7pd8jPF/+2vkqSS\nt9/XobNHhrlFANDxKNY6gBEi8s/v9NG2HyQ1HCISY5HqRIgIAMCkCBEBAAPFWjvyDxEpLnfLYpFi\nD3n05jPTjHum3JYjr/XYf4YxQ3pq3FknqVtSAqtpAABTsf3hZdlvv1mSdPD2O1V1z/1hbhEAhBfF\nWjsKDhHJbSREJDUpQWf2c7KSBgAwJUJEAKC+FhVrW7Zs0W233abTTz9dktSnTx/94he/0Pz58+X1\neuV0OvXEE08oPj6+TRvbmVRU1WrrrgLjtf+2x6fG3arNAzKN1ylJ8Vo8a4TsiebtLwCAOQWGiKSr\n6Iv/hLlFABA5WryydtZZZ+mZZ54xXt9zzz3Kzs7WhAkT9OSTTyonJ0fZ2dlt0sjOxLf18bNdLpVW\n1mrMl3/Rne8+ZVwPFSIyvF8GhRoAwHS6XTpJ8R//RRIhIgAQSpttg9yyZYsefPBBSdLYsWO1cuVK\nUxVrRojIv/boo7y9kpoIEamTHMnHQkQAANGB3SfNQIgIADRLi4u13bt368Ybb1RZWZluvvlmVVdX\nGwNPWlqaXC5XE98QHfxDRIrK3YqxSLFej958+liIyOS5b+hwzLGwkDFDe2nciBMJEQGAKMXuk4YF\nhIjMvVNV9xIiAgANaVGxdvLJJ+vmm2/WhAkTtGfPHl199dXyer3G9bq6umZ9T2pqomJjW1+sOJ32\nVn9HS63YsCMgROStRkJEHMkJ+ungXpo1aYCs1vYPEQlnv0Qy+iU0+iU0+iU0+uX4mH33iU9AiMjX\n36muW0oYWwMAka9FxVr37t01ceJESdJJJ52k9PR07dixQzU1NbLZbDpw4IAyMjKa/J6SkqqW/PEB\nnE67XK6KVn9PS7g9Xv19+17jtf+2x0d+Nl//6DPKeJ2alKAHZh4JESkuPtjubQtnv0Qy+iU0+iU0\n+iW0lvaLmQo8dp8Esv5ntxwjh0mSDjscKtr13/A2CAA6iRYVa7m5uXK5XJo9e7ZcLpeKiop0ySWX\naOPGjZo8ebLef/99jR49uq3bGnHKKt0qLndrxDf/1P1vPWy8HypE5Mx+TkJEAMAEwr37JOKK4qws\n6cMPj/z88ceK+elP5QxjcyKufyIIfdMw+qZx9E/DWts3LSrWMjMzdeedd+rDDz+Ux+PR4sWL1b9/\nf919991av369evbsqSlTpjT9RZ2ML0TEd9asW1KCcv1W04q6OjTzhpWSjoaISHLYCREBADMJ5+6T\ncO828R8j5fHI2SvNuG6EiIRxtZrV8obRNw2jbxpH/zSsuX3TWEHXomItKSlJy5cvr/f+qlWrWvJ1\nEc8/RKS43C1HcoLO/Lz6INUAABKzSURBVHGqbrn6WMRwvRCRIT017qyTCBEBAJMx2+6TUGPkFT/8\nQxe+8JAkQkQAoDXaLLo/mq3fvDsgROSZR6crueZYlfz0uq1KzS9USUWNUv1W0qwx7R8iAgCILGbb\nfRI8Rq5ePMH4mRARAGgdirUmVFTV6rNdxw6C+4eILL3iIc14/GZlx1l16ZhTA7d/AEAYvPPO2/rP\nf77RzTfPDXdTTMtMu0/cHq+25R8ZI+3V5Vr33NWSpHKbXbcueF1LEu1KCGcDAaABnWW8pFhrgG9b\nx9ZdBSqtrNXAPTv1yOsLjeuT5m1QjEUaX+lWRmqiEuKsykhNDGOLAQDoWL6grbFffKR57z0tSbp7\nxsP6otdPFFNRo7KjYyQAoGUo1oL4Dkhv/Od3+mjbD5ICV9N29hqge2b8WpKUarepWxJzhgAiy/79\nP+imm36hXr16a8eOf+viiy/VN9/s1hdf7NTFF0/TpZdOD3cTESW6dYnVypeuV3pZgT47eZgWX3Ls\nbBpjJIBI1xnGS4q1o4IPSFssUsxhr9767aXGPcEhIkP7pLPlEUBE+vrrfD3yyG9UXl6uq66artdf\nz1Vtba3uu29+RAw+6Pxid2yX84IjQSn3TX1Q/z5pcMB1xkgAnUGkj5cUa0cFH5C+eeOzumjnJuO1\n/7PTUpMSdGY/J3H8AJqUet7Zit31ZZt936F+/VXy1y1N3terV29165aiuLh4paY65HRmqKqqSgcP\nVrZZW2BeSXfcqi6vrJYk7f/vfmX843ulhQjaAoDmCNdYKUX+eEmxpiMhIlt3FRiv/bc9zr1iqb7p\nfqrxOiUpXotnjeAB1wCapbmDRVuzWq0hf27uA5mBUCzFRUrvd4okqXLhg6q+9XZZJWVn9SFoC0CL\nhWuslCJ/vDR1sebb+vjZLpdKK2vVs2Svnl81x7juv5rmM7xfBoUaAMB0EtavU/ItN0qSij7bqcMn\nnhR4naAtAGhzpizWjBCRf+3RR3l7JUl/XJatJHeVJGnl6Gv05oiLJUkxFqmuTnIks60DAGBCXq8c\nI86Q9fs9qh17gcpe/ZNksYS7VQBgCpa6MK7xuVwVTd/UBKfT3uzv8Q8RKSp3HynEDh9W7lOXGPcE\nh4iMHdZL40ac2Om2dRxPv5gJ/RIa/RIa/RJaS/vF6bS3Q2uiV0v7uC1/Z2N3bFfq0RCR0pxcec47\nv82+Oxz4N90w+qZh9E3j6J+GNbdvGhsfTbWyFhwiMnz3P7XorYclSSWJ3XT1jWuMaylJ8RreL0Mz\nMk+TNSamw9sKAEA4Jd05V11eXilJcn1XINlsYW4RAJiPaYq1xkJErrl+pYqTHMbr1KQEQkQAAKYU\nKkQEABAeUV+sBYeIdKmt1mvPXm5cDxUicmY/J4UaAMB0mgoRAQB0rKgt1kKFiIzc/anuy31UkvTL\nmc/qe0dvSUdDRCQ5eDYMAMCM/ENEzs9U2fo3CREBgAgQdcVaqBARS92xEJFvMn6suVcsDRiExgzp\nqXFnndTpQkQAAGitgBCR19+SZ8zYMLcIAOATdcVacIjImbv/qfuPhoi8OOZavXXmZOMaISIAADML\nCBH59oDUpUuYWwQA8BdVxZrb49W2fJfxmhARAAht9+6vFR8fr5NO+lGT9xYVFeqll57X/Pn3dUDL\n0BEsJcVK73uyJKly4WJV3zovvA0CgAh0PGOlv48+2qSxY7PapA1RtZxUVulWcblbXWqrAwq1SfM2\nBBRqEiEiACKH2+NVQUmV3B5vh/2Zf/nLZu3Z812z7k1LS6dQiyIJr79qFGpFn+2kUAPQKUT6WOnj\n8Xi0fv26NmtDp15Zc3u82ld4UF6PVwlxVnVLSlDW3s906/qHJBEiAiCy+Z+xLS53y5GcoKF9nK3a\nmn3ddddo8eJfq1ev3iooOKAFC+7QypVrA+755pvdeuutP+kvf9ms1NRU/epXizRy5LlKTU3VqFGj\n9eSTjyk2NlYxMTF66KFHdfDgQS1ceLdeeukVzZgxRdnZl+v99zeptrZWTz/9eyUmdm2L7kAb8wVt\nGeexvV45zh4i63ffqnbMWJW9toEQEQARL1LGSo/Ho+efX6bY2FhlZHTX3XcvlNvt1v33L1Btba08\nHo/mzbtb//M/b+mbb3brN795VI899utW//07ZbEW8B+twi2HPUFDT0/XTY/9Qrf+ezshIgA6heAz\ntkXlbuN1dlafFn3n+PET9eGH7+vqq2fpb3/7q7KyxtW759RTT9PZZ5+j88+/QD/5yUAdOnRII0eO\n0siRo/Svf32q22+/S3369NOLLy7X+++/q3PPPc/4rNfr1Y9//GMtW7ZCDzxwj7Zu/ZfOO+/8FrUV\n/3979x/T5J3HAfxdWn4cQqVIa86fWUiUzADioVFh/oiIy9iW3DzQYOe2m1GHZmbGCTRMZ+bYUGYk\nmm1sYubINjFlc97ipiGRxdwqiL1UZTMebvEERahigQpo4Xt/GKvYpwxq4Wnt+/Vff4Af3y28+fZ5\nvu3wkPrDZnHITawwrADANxEhIv/iK1352mvZKCn5BGr1aHz8cQlOnKhCaGgotFod8vO3oKmpEVeu\n/A/Z2S/j11/PY9OmPM/+w4/wy9Mg7z9oN9p7IAQQevl3bFgxE8FnLWjbX44fdh/CmNF/QZACGKMO\nQ1ryBGQvngKdJpwLNSLyCY/usX3Yfy5aPT7NIy1tCX7++QQA4JdfTmLxYtcCkvL009MAABrNGJSW\nfoz161ejquoYbDaby32Tk5MBAFrtWNjtnR7NScOnX0cCyPq2xLlQa718nQs1IvIbvtKVN2/eQGPj\nFRgMb2P9+tUwm8+gtbUF06YloL7+HHbuLERTUyNmz57r0TwD8bsja48+aCtPliPzdCUAYG3+tyhY\nshDZwUosnR/b//QPIiIfcn+PrZS2jm7YOnug04QP+fuOHh0FnU6H336rR1+fgFarG9TXqVTBAICS\nkmKsWPEKZs+ei6+/LkdX122X+yqVSgB9AAAhxJBnpOHTfcfh7MiIrg5888nLAIADqXqcSNNjuyoE\noXIOSEQ0BL7SlSpVMGJitNi79zOX27744huYzXX47jsj6uvP4dlnM4Y8z0D87sjaww+asteBzNOV\nMCb/HS9sPIxrd4Jg67x3W2iwkkfSiMhnjY4IRbRa+s9mTWQYRkd4/if1kiXPYdeuIixcuMjtfRQK\nBXp7XV+RtNluYfz4Cbhz5w5Onfo3HA6Hx3PQyGtrf9CRz547DgD456rPYJz1D+cfNkRE/sJXulKt\nVgMA/vjjdwCA0XgQDQ3/xenTNTh9ugazZs3GW2+9jQsXfoVCESTZr57yu8Xaww9ar1KFFzYexoF5\nrwB4/AeNiGikhAYrkTRFK3lb0pSYx3qhKSVlHhobG7FggfsCSkxMwu7dO1FXV9vv+qVLlyE/fxPe\neScXS5cuw48//oDOTp7q6C806gcdaZy1FC9sPIxW9b1XjNmRRORvfKkr8/K2oLBwG3JyVuHsWQsm\nTZqMCRMm4ssv92P9+tXYvn0rsrNXIiYmBg7HXRQU5Ho828MUQsZzWFpbOzz6uq+rLvbbaHhfWvIE\njzcaPmm02kiP832SMRdpzEXacOfy4I0grGjr6IbmoXer9fQdrgDAbK7D0aP/QkHBNi9O+4CnuWi1\nkcMwzZPL04xLvjnDjnSDv+vcYzbuMZuBsSvdG2w2A/Wj1/esFRYWwmKxQKFQwGAwICEhwdv/hPNt\n96UeNCIif6EMCkJ22hSv7rEtKytFTY0J77+/A83Nzdi+fYvLfZKS/obXX1/zWP8O+S52JBE9SQK9\nK716ZK22thZlZWUoLS3FpUuXYDAYUFFR4fb+j7sK77nbC2VIMHrv3OXetEfwVSBpzEUac5HGXKTx\nyNrI8DTj+1/n8jlrxJ/pATAb95jNwJiPe944subVPWsmkwlpaWkAgNjYWNhstmHd6xAarMRfY0ax\nhIiIiB7BN9oiIvJ/Xl2sWa1WaDQa5+Xo6Gi0tkp/NgIREVEgKSwsxLJly7B8+XKcPXtW7nGIiMgP\nDOvnrP3ZGZYaTThUqsd/xY+n1khjLtKYizTmIo25SGMuQ1NbW4vLly+joqJiUNsEiIiIAC8v1nQ6\nHaxWq/NyS0sLtFrpt9sEgLY21w9bHSqeJyuNuUhjLtKYizTmIo171obO3TaBiIgImScjIiJf5tXT\nIFNSUnDs2DEAQH19PXQ6HYuIiIgCHrcJEBGRJ7x6ZG3GjBmYNm0ali9fDoVCga1bt3rz2xMRET0R\nBvNGzJ5uFQjkI5iDwXzcYzbuMZuBMR/3Hjcbr+9Z27Rpk7e/JRERkV8b6jYBwLOtAjx1d2DMxz1m\n4x6zGRjzcc/n3rqfiIiIXHGbABERecKrH4pNRERE0oqLi1FXV+fcJhAXFyf3SERE5OO4WCMiIiIi\nIvJBPA2SiIiIiIjIB3GxRkRERERE5IO4WCMiIiIiIvJBXKwRERERERH5IC7WiIiIiIiIfJDXPxR7\nJBUWFsJisUChUMBgMCAhIUHukWSzY8cOnDlzBg6HA2vWrEF8fDw2b96M3t5eaLVa7Ny5EyEhIXKP\nKYvu7m48//zzyMnJwZw5c5gLgCNHjmDfvn1QqVR48803MXXq1IDPxW63Izc3FzabDXfv3sW6deug\n1Wrx7rvvAgCmTp2Kbdu2yTvkCLp48SJycnLw6quvQq/X49q1a5LPkSNHjuDAgQMICgpCVlYWMjMz\n5R49YLETXbEbB8Z+lMaOlMaelDbsfSn8VE1NjVi9erUQQoiGhgaRlZUl80TyMZlMYtWqVUIIIW7e\nvCnmz58v8vLyxNGjR4UQQnz00Ufiq6++knNEWe3atUu89NJLorKykrmIe8+R9PR00dHRIa5fvy4K\nCgqYixCivLxcFBcXCyGEaG5uFkuWLBF6vV5YLBYhhBAbN24U1dXVco44Yux2u9Dr9aKgoECUl5cL\nIYTkc8Rut4v09HTR3t4uurq6REZGhmhra5Nz9IDFTnTFbvxz7EdX7Ej32JOuRqIv/fY0SJPJhLS0\nNABAbGwsbDYbOjs7ZZ5KHjNnzkRJSQkAQK1Wo6urCzU1NVi0aBEAYOHChTCZTHKOKJtLly6hoaEB\nCxYsAADmgns/O3PmzEFERAR0Oh3ee+895gJAo9Hg1q1bAID29nZERUWhqanJeXQikHIJCQnB559/\nDp1O57xO6jlisVgQHx+PyMhIhIWFYcaMGTCbzXKNHdDYia7YjQNjP0pjR7rHnnQ1En3pt4s1q9UK\njUbjvBwdHY3W1lYZJ5KPUqlEeHg4AMBoNGLevHno6upyHqIfM2ZMwGZTVFSEvLw852XmAjQ2NqK7\nuxtr165FdnY2TCYTcwGQkZGBq1evYvHixdDr9di8eTPUarXz9kDKRaVSISwsrN91Us8Rq9WK6Oho\n530C+few3NiJrtiNA2M/SmNHuseedDUSfenXe9YeJoSQewTZVVVVwWg0Yv/+/UhPT3deH6jZHD58\nGNOnT8fEiRMlbw/UXADg1q1b2Lt3L65evYqVK1f2yyJQc/n+++8xbtw4lJWV4cKFC1i3bh0iIyOd\ntwdqLlLcZcGMfAcfiwfYja7YjwNjR0pjTw6dN/rSbxdrOp0OVqvVebmlpQVarVbGieR18uRJfPrp\np9i3bx8iIyMRHh6O7u5uhIWF4fr16/0OzwaK6upqXLlyBdXV1WhubkZISAhzwb1XeZKSkqBSqTBp\n0iSMGjUKSqUy4HMxm81ITU0FAMTFxaGnpwcOh8N5e6Dmcp/Uz47U7+Hp06fLOGXgYidKYzdKYz+6\nx450jz05ON7uS789DTIlJQXHjh0DANTX10On0yEiIkLmqeTR0dGBHTt2oLS0FFFRUQCAuXPnOvM5\nfvw4nnnmGTlHlMXu3btRWVmJQ4cOITMzEzk5OcwFQGpqKk6dOoW+vj60tbXh9u3bzAXA5MmTYbFY\nAABNTU0YNWoUYmNjUVdXByBwc7lP6jmSmJiIc+fOob29HXa7HWazGcnJyTJPGpjYia7Yje6xH91j\nR7rHnhwcb/elQvjxMcvi4mLU1dVBoVBg69atiIuLk3skWVRUVGDPnj146qmnnNd9+OGHKCgoQE9P\nD8aNG4cPPvgAwcHBMk4prz179mD8+PFITU1Fbm5uwOdy8OBBGI1GAMAbb7yB+Pj4gM/FbrfDYDDg\nxo0bcDgc2LBhA7RaLbZs2YK+vj4kJiYiPz9f7jFHxPnz51FUVISmpiaoVCqMHTsWxcXFyMvLc3mO\n/PTTTygrK4NCoYBer8eLL74o9/gBi53YH7txcNiPrtiR0tiTrkaiL/16sUZERERERPSk8tvTIImI\niIiIiJ5kXKwRERERERH5IC7WiIiIiIiIfBAXa0RERERERD6IizUiIiIiIiIfxMUaERERERGRD+Ji\njYiIiIiIyAdxsUZEREREROSD/g+hEI3UbpA2cQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f7c69246710>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "xAP1EoQi86XB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Inference"
      ]
    },
    {
      "metadata": {
        "id": "K2yfNk3d8-Vj",
        "colab_type": "code",
        "outputId": "aef548b1-d0a1-4481-a9fe-4c975f278a3f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        }
      },
      "cell_type": "code",
      "source": [
        "# Feed in your own inputs\n",
        "X_infer = np.array((0, 1, 2), dtype=np.float32)\n",
        "standardized_X_infer = X_scaler.transform(X_infer.reshape(-1, 1))\n",
        "pred_infer = (lm.predict(standardized_X_infer) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\n",
        "print (pred_infer)\n",
        "df.head(3)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[10.00356575 13.65348546 17.30340518]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>10.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>13.65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>17.30</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     X      y\n",
              "0  0.0  10.00\n",
              "1  1.0  13.65\n",
              "2  2.0  17.30"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "metadata": {
        "id": "PHH0fYp_BYC5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Interpretability"
      ]
    },
    {
      "metadata": {
        "id": "OhXo8CbPBZ-G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies it's importance/impact on the output variable y. We can interpret our coefficient as follows: By increasing X by 1 unit, we increase y by $W$ (~3.65) units. \n",
        "\n",
        "**Note**: Since we standardized our inputs and outputs for gradient descent, we need to apply an operation to our coefficients and intercept to interpret them. See proof below."
      ]
    },
    {
      "metadata": {
        "id": "JZxnrDuCBbK9",
        "colab_type": "code",
        "outputId": "adbe06d5-449a-4aa0-ebbb-ef5fdd5ef9e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Unstandardize coefficients \n",
        "coef = lm.coef_ * (y_scaler.scale_/X_scaler.scale_)\n",
        "intercept = lm.intercept_ * y_scaler.scale_ + y_scaler.mean_ - np.sum(coef*X_scaler.mean_)\n",
        "print (coef) # ~3.65\n",
        "print (intercept) # ~10"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[3.64992275]\n",
            "[10.00356788]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yVmIP13u9s33",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Proof for unstandardizing coefficients:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "ViDPSLbR9v4B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that both X and y were standardized.\n",
        "\n",
        "$\\frac{\\mathbb{E}[y] - \\hat{y}}{\\sigma_y} = W_0 + \\sum_{j=1}^{k}W_jz_j$\n",
        "\n",
        "$z_j = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n",
        "\n",
        "$ \\hat{y}_{scaled} = \\frac{\\hat{y}_{unscaled} - \\bar{y}}{\\sigma_y} = \\hat{W_0} + \\sum_{j=1}^{k} \\hat{W}_j (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n",
        "\n",
        "$\\hat{y}_{unscaled} = \\hat{W}_0\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} \\hat{W}_j(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j + \\sum_{j=1}^{k}(\\frac{\\sigma_y}{\\sigma_j})x_j $\n"
      ]
    },
    {
      "metadata": {
        "id": "rToCXKqeJcvj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Regularization"
      ]
    },
    {
      "metadata": {
        "id": "L4GFv8xRJmOZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Regularization helps decrease over fitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties. \n",
        "\n",
        "**Note**: Regularization is not just for linear regression. You can use it to regualr any model's weights including the ones we will look at in future lessons."
      ]
    },
    {
      "metadata": {
        "id": "D_OcpRxF-Oj7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* $ J(\\theta) = = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}\\sum\\sum W^2$\n",
        "* $ \\frac{\\partial{J}}{\\partial{W}}  = X (\\hat{y} - y) + \\lambda W $\n",
        "* $W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "where:\n",
        "  * $\\lambda$ is the regularzation coefficient"
      ]
    },
    {
      "metadata": {
        "id": "HHaazL9f8QZX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize the model with L2 regularization\n",
        "lm = SGDRegressor(loss=\"squared_loss\", penalty='l2', alpha=1e-2, \n",
        "                  max_iter=args.num_epochs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VTIUZLbGZP4e",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "e284d26b-6091-4ce9-b40c-5b9d675f0837"
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "lm.fit(X=standardized_X_train, y=standardized_y_train)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGDRegressor(alpha=0.01, average=False, epsilon=0.1, eta0=0.01,\n",
              "       fit_intercept=True, l1_ratio=0.15, learning_rate='invscaling',\n",
              "       loss='squared_loss', max_iter=100, n_iter=None, penalty='l2',\n",
              "       power_t=0.25, random_state=None, shuffle=True, tol=None, verbose=0,\n",
              "       warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "ORwkUqcuZhbX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Predictions (unstandardize them)\n",
        "pred_train = (lm.predict(standardized_X_train) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\n",
        "pred_test = (lm.predict(standardized_X_test) * np.sqrt(y_scaler.var_)) + y_scaler.mean_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IWCvYxBxZhd5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a38bb230-eb97-43c1-d43d-8dead30dedcb"
      },
      "cell_type": "code",
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((y_test - pred_test) ** 2)\n",
        "print (\"train_MSE: {0:.2f}, test_MSE: {1:.2f}\".format(\n",
        "    train_mse, test_mse))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 1.09, test_MSE: 1.15\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mdNX2W5eh2ma",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Regularization didn't help much with this specific example because our data is generation from a perfect linear equation but for realistic data, regularization can help our model generalize well."
      ]
    },
    {
      "metadata": {
        "id": "C2mrVS4UZp3Q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "5ba6159b-0791-4093-9ed9-8925386edf36"
      },
      "cell_type": "code",
      "source": [
        "# Unstandardize coefficients \n",
        "coef = lm.coef_ * (y_scaler.scale_/X_scaler.scale_)\n",
        "intercept = lm.intercept_ * y_scaler.scale_ + y_scaler.mean_ - (coef*X_scaler.mean_)\n",
        "print (coef) # ~3.65\n",
        "print (intercept) # ~10"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[3.61386358]\n",
            "[11.67635438]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "V74lNFE5v5pQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Categorical variables"
      ]
    },
    {
      "metadata": {
        "id": "2r6Xhyg7v5vX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In our example, the feature was a continuous variable but what if we also have features that are categorical? One option is to treat the categorical variables as one-hot encoded variables. This is very easy to do with Pandas and once you create the dummy variables, you can use the same steps as above to train your linear model."
      ]
    },
    {
      "metadata": {
        "id": "unhcIOfMxQEQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "ecf36ce0-28af-4381-b61b-2592a3c1b289"
      },
      "cell_type": "code",
      "source": [
        "# Create data with categorical features\n",
        "cat_data = pd.DataFrame(['a', 'b', 'c', 'a'], columns=['favorite_letter'])\n",
        "cat_data.head()"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>favorite_letter</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>b</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>c</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  favorite_letter\n",
              "0               a\n",
              "1               b\n",
              "2               c\n",
              "3               a"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "metadata": {
        "id": "m4eQmJdrxQGr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "247aaac2-afcb-4899-e415-91d3fe169fbf"
      },
      "cell_type": "code",
      "source": [
        "dummy_cat_data = pd.get_dummies(cat_data)\n",
        "dummy_cat_data.head()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>favorite_letter_a</th>\n",
              "      <th>favorite_letter_b</th>\n",
              "      <th>favorite_letter_c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   favorite_letter_a  favorite_letter_b  favorite_letter_c\n",
              "0                  1                  0                  0\n",
              "1                  0                  1                  0\n",
              "2                  0                  0                  1\n",
              "3                  1                  0                  0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "B5R8x-KyiBWJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now you can concat this with your continuous features and train the linear model."
      ]
    },
    {
      "metadata": {
        "id": "eVOXoCRsokzp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TODO"
      ]
    },
    {
      "metadata": {
        "id": "4c7ttuUwfeLA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "- polynomial regression\n",
        "- simple example with normal equation method (sklearn.linear_model.LinearRegression) with pros and cons vs. SGD linear regression"
      ]
    }
  ]
}
